share
As your IT infrastructure grows in size and becomes complex, you may be forced to spend more attention ensuring that everything is managed properly.
Since the rise of the cloud, this has become especially more relevant, given that we now have an abundance of technical information spread across many different locations.
But being able to carry out comprehensive log analysis can be tedious, time-consuming and painfully boring.
A simple yet effective way to liberate yourself from this burden is to use software or applications that are designed to make monitoring logs easy… and that’s exactly what Alexis Gardin has done in his application, Logub.
Unlike many other SaaS solutions, Logub can carry out these actions on-premises and open source. By following the steps below, you’ll be able to create an application that will collect, explore, and analyze application logs for you.
At the heart of this application was a reliance on RediSearch’s ability to explore and analyze logs across different locations with unrivaled efficiency. Let’s look into how Alexis brought this application together.
But before we dive in, we should let you know that we also have an exciting range of applications for you to check out on the Redis Launchpad.
You’ll build a special app that’s powered by Redis to collect, analyse and explore log applications. In the steps below, we’ll go through A-Z of what’s required to build this application, along with the required components.
Even if you’re new to Redis, we’ll show you exactly how to get to grips with this powerful database. Are you ready to get started?
Ok, let’s dive straight in.
In the demo, a DEMO app will publish logs in Logub. You’ll be able to interact with this DEMO app to generate your logs. You’ll then be able to request them in Logub
Firstly, make sure that the given ports are open on your system: 8080, 8081, 3000 & 6379.
git clone https://github.com/redis-developer/logub
Stop testing, start deploying your AI apps. See how with MIT Technology Review’s latest research.
docker-compose up -d
% docker-compose ps
NAME COMMAND SERVICE STATUS PORTS
demo_fluentd_1 "tini -- /bin/entryp…" fluentd running 5140/tcp, 0.0.0.0:24224->24224/tcp, :::24224->24224/tcp, 0.0.0.0:24224->24224/udp, :::24224->24224/udp
demo_logub-controller_1 "java -XX:+UnlockExp…" logub-controller running 0.0.0.0:8080->8080/tcp, :::8080->8080/tcp
demo_logub-generator_1 "java -XX:+UnlockExp…" logub-generator running 0.0.0.0:8081->8081/tcp, :::8081->8081/tcp
demo_logub-ui_1 "docker-entrypoint.s…" logub-ui running 0.0.0.0:3000->3000/tcp, :::3000->3000/tcp
demo_redis_1 "redis-server /usr/l…" redis running 0.0.0.0:6379->6379/tcp, :::6379->6379/tcp
Go to localhost:3000 to explore logs.
It may take around one minute for you to see the logs coming. You can view the logs on the page by filtering them with the sidebar on the right. Alternately, you search for them by filter or full-text query via the search bar at the top.
When you click on a log, the details will be displayed and you’ll have the option to index business properties. These business properties can be used as filters afterwards.
Next, go to localhost:3000/demo to access the playground and add your custom logs. This demo page will allow you to:
If you return to the main page, you can try to search for the logs you’ve just generated.
Note: There is a latency of around 1 minute between the production of a log in a container and its display in Logub. This latency is caused by the process of collecting, formatting, and ingesting the logs into the database.
Data is stored using the Fluentd Redis Plugin. It stores each log with HSET. For example: HSET level DEBUG message “Hello World” thread main
public class LogubLog {
@Builder.Default
private String id = UUID.randomUUID().toString();
@Builder.Default
private String index = "principal";
@NonNull
private SystemProperties systemProperties;
@Builder.Default
private Map<String, Object> businessProperties = Collections.emptyMap();
@Builder.Default
private Optional<String> message = Optional.empty();
@Builder.Default
private Instant timestamp = Instant.now();
@Builder.Default
private Optional<String> service = Optional.empty();
@Builder.Default
private Optional<String> logger = Optional.empty();
@Builder.Default
private Optional<String> thread = Optional.empty();
@Builder.Default
private Optional<String> source = Optional.empty();
@Builder.Default
private LogLevel level = UNKNOWN;
}
public class SystemProperties {
@Builder.Default
Optional<String> imageName = Optional.empty();
@Builder.Default
Optional<String> containerName= Optional.empty();
@Builder.Default
Optional<String> containerId= Optional.empty();
@Builder.Default
Optional<String> env= Optional.empty();
@Builder.Default
Optional<String> host= Optional.empty();
}
This is the object that you’ll use to manipulate logs and retrieve them from Redis. Carrying out this action will allow the Logub UI to display the logs. To make complex queries in your logs, you can use RediSearch.
Since you’re able to change the RediSearch schema dynamically, you can use the ‘List’ data structure to keep track of which schema is indexed.
public class LogSearch {
@Builder.Default
private List<LogubFieldSearch> texts = emptyList();
@Builder.Default
private List<LogubFieldSearch> systemProperties = emptyList();
@Builder.Default
private List<LogubFieldSearch> businessProperties = emptyList();
@Builder.Default
private List<LogubFieldSearch> basicProperties = emptyList();
@Builder.Default
private List<LogubFieldSearch> levels = Collections.emptyList();
@Builder.Default
private int limit = 25;
@Builder.Default
private int offset = 0;
@Builder.Default
private Optional<LogubSort> sort = Optional.empty();
@Builder.Default
private Instant beginAt = Instant.now().minus(15, ChronoUnit.MINUTES);
@Builder.Default
private Instant endAt = Instant.now();
@SneakyThrows
public QueryBuilder toQuery() {
var query = new QueryBuilder();
var businessPrefix = "businessProperties.";
var systemPropertiesPrefix = "systemProperties.";
for (LogubFieldSearch properties : businessProperties) {
query.append(QueryBuilders.tag(businessPrefix + properties.getName(), properties.getValues(),
properties.isNegation()));
}
for (LogubFieldSearch properties : systemProperties) {
query.append(QueryBuilders
.tag(systemPropertiesPrefix + properties.getName(), properties.getValues(),
properties.isNegation()));
}
for (LogubFieldSearch properties : basicProperties) {
query.append(QueryBuilders
.tag(properties.getName(), properties.getValues(),
properties.isNegation()));
}
if (!levels.isEmpty()) {
for (LogubFieldSearch level : levels) {
var onError = !level.getValues().stream().allMatch(v -> Arrays.stream(LogLevel.values())
.anyMatch(enumLevel -> enumLevel.name().equalsIgnoreCase(v)));
if(onError){
log.error("bad payload for levels {}", level);
throw new IllegalArgumentException("bad payload for level");
}
query.append(QueryBuilders.tag("level",level.getValues(), level.isNegation()));
}
}
for (LogubFieldSearch text : texts) {
if(!text.getType().equals(LogubFieldType.FullText)){
log.warn("type {} not handle for text search", text.getType());
}
for (String value : text.getValues()) {
query.append(QueryBuilders.text("message", value, text.isNegation()));
}
}
return query;
}
}
The above command will allow you to create a plain text RediSearch query based on the user input. There are a bunch of small QueryBuilders built on the top of the RediSearch library. It’s the same command that will be sent by Logub UI to carry out a comprehensive and efficient search in your logs.
You can carry out the search by tag or full-text search. Below are some examples you can use as models if you get stuck.
For you to have a good testing experience with the app, we highly recommend that you create your own log with the playground, add business properties and then do some experimentation just to get a feel for it.
For now, Logub can handle only one particular log format. In the future, this format will be extended and more customizable.
Here’s the Logub format
{
"level": "....",
"thread": "....",
"logger": "....",
"message": "....."
}
Please note, these fields are not mandatory.
If you want to add your business properties, you’ll need to add a nested JSON object that has “mdc” as the key. For example:
{
"level": "....",
"thread": "....",
"logger": "....",
"message": ".....",
"mdc":{
"myproperties": "....",
"anotherOne":".....",
}
}
To explore logs in Loghub, your containers need to use the Docker Fluentd logging driver. Here’s a configuration example for customer integration.
# Your container(s)
MY-CUSTOM-APP:
image: "MY-CUSTOM-APP-IMAGE"
logging:
driver: "Fluentd"
options:
Fluentd-address: localhost:24224
tag: MY-CUSTOM-TAG
## Logub Fluentd + Redis conf
logub-controller:
image: "logub/logub-controller:0.1"
ports:
- "8080:8080"
depends_on:
- redis
links:
- "Fluentd"
- "redis"
logub-ui:
image: "logub/logub-ui:0.1"
ports:
- "3000:3000"
depends_on:
- redis
Fluentd:
image: "logub/logub-Fluentd:0.1"
links:
- "redis"
ports:
- "24224:24224"
- "24224:24224/udp"
redis:
image: "redislabs/redismod"
command: ["/usr/local/etc/redis/redis.conf","--bind","redis","--port", "6379"]
volumes:
- ./redis/data:/data
- ./redis/redis.conf:/usr/local/etc/redis/redis.conf
ports:
- "6379:6379"
RediSearch
Logub uses the functionality of RediSearch to process application logs. When logs are persisted in the Redis database, they’re accompanied by 3 types of fields:
{
"timestamp": "2021-05-14 11:01:11.686",
"level": "WARN",
"thread": "scheduling-1",
"mdc": {
"app": "Toughjoyfax",
"correlationId": "521f075f-36be-4f85-957e-d1c87ad71aa8",
"originRequest": "Tonga",
"origin": "LoremIpsum"
},
"logger": "com.loghub.loggenerator.service.LoggerService",
"message": "Doloremque dolores ut minima sed."
}
Here we have an example of a log that describes how our tool functions when Fluentd flattens and persists in the Redis database. The Logub API will allow the user or the company to index one or all fields of the mdc object.
In this project, the Tag Datatype is widely used. Logs are often searched based on business properties when searching in logs (eg a customer id). Moreover, we also use the TextField Datatype for log messages. This allows the user to do a full-text search in this field.
Here’s a simplified schema of the search process:
Redis
Redis is used to store logs by Fluentd like this in the HashSet type of Redis.
To keep track of the indexed field by the user, you can also add a ‘schema’ object which uses the List type of Redis.
A drawback to the explosion of digital innovations, such as the cloud, is that monitoring IT infrastructure can become complex. Applications are likely to be scattered across different locations, making it challenging to difficult to pull every log into one location.
However, by using Redis, the monotony of this process is removed, allowing you to search, gather and store logs with ease. If you want to learn more about the ins and outs of how this application was made, then you can check out Alexis’ YouTube video here.
We also have an exciting range of innovative applications that have been developed by talented programmers just like Alex on our Launchpad. Check it out, be inspired and see what innovations you can come up with using Redis.
Who built this application?
Alexis Gardin
Alexis is an innovative software engineer who currently works for Zendoc. Head over to his GitHub page to see what other projects he’s been involved in.
