Splunk Commands - The Ultimate Guide

Goal of Search with Splunk

The goal of search is to help you find exactly what you need. It can mean filtering, summarizing, and visualizing a large amount of data, to answer your questions about the data.

Splunk Installation is the first step to the goal of searching with Splunk. The Summary dashboard gives you a quick overview of the data visible to you. Click the Launch search app on the Splunk Welcome tab. If you’re on the Splunk Home tab, click Search under Your Apps.

Few points about this dashboard:

• The search bar at the top is empty, ready for you to type in a search.
• The time range picker to the right of the search bar permits time range adjustment. You can see events from the last 15 minutes, for example, or any desired time interval. For real-time streaming data, you can select an interval to view, ranging from 30 seconds to an hour.
• The All indexed data panel displays a running total of the indexed data.

The next three panels show the most recent or common values that have been indexed in each category:

• The Sources panel shows which files (or other sources) your data came from.
• The Source types panel shows the types of sources in your data.
• The Hosts panel shows which host your data came from.

Search navigation menus near the top of the page include:-

• The summary is where we are.
• Search leads to the main search interface, the Search dashboard.
• Status lists dashboards on the status of your Splunk instance.
• Dashboards & Views list your dashboards and views.
• Searches & Reports lists your saved searches and reports.

The Search Dashboard

If you click the Search option or enter a search in the search bar, the page switches to the Search dashboard (sometimes called the timeline or flashtimeline view). When a search is kicked off, the results almost immediately start displaying. For example, entering an asterisk (*) in the search bar retrieves all the data in your default indexes.

The contents of this dashboard:-

• Timeline: A graphic representation of the number of events matching your search over time.
• Fields sidebar: Relevant fields along with event counts. This menu also allows you to add a field to the results.
• Field discovery switch: Turns automatic field discovery on or off. When Splunk executes a search and field discovery is on, Splunk attempts to identify fields automatically for the current search.
• Results area: This shows the events from your search. Events are ordered by Timestamp, which appears to the left of each event. Beneath the Raw text of each event are any fields selected from the Fields sidebar for which the event has a value.

When you start typing in the search bar, context-sensitive information appears below, with matching searches on the left and help on the right.

The search job controls are only active when a search is running. If you haven’t run a search, or if your search has finished, they are inactive and greyed out. But if you’re running a search that takes a long time to complete, you can use these icons to control the search progress:

• Sending a search to the background lets it keep running to completion on the server while you run other searches or even close the window and log out. When you click Send to background, the search bar clears and you can continue with other tasks. When the job is done, a notification appears on your screen if you’re still logged in; otherwise, Splunk emails you (if you’ve specified an email address). If you want to check on the job in the meantime, or at a later time, click the Jobs link at the top of the page.

• Pausing a search temporarily stops it and lets you explore the results to that point. While the search is paused, the icon changes to a play button. Clicking that button resumes the search from the point where you paused it.

• Finalizing a search stops it before it completes, but retains the results to that point and so you can view and explore it in the search view.

• In contrast, canceling a search stops it from running, discards the results, and clears them from the screen.

Watch this Splunk Tutorial for Beginners video:

The Job inspector icon takes you to the Job inspector page, which shows details about your search, such as the execution costs of your search, debug messages, and search job properties.
Use the Save menu to save the search, save the results, or save and share the results. If you save the search, you can find it on the Searches & Reports menu. If you save the results, you can review them by clicking on Jobs in the upper right corner of the screen.
Use the Create menu to create dashboards, alerts, reports, event types, and scheduled searches. Moving down to the upper left corner of the Results area, you see the following row of icons.

By default, Splunk shows events as a list, from most recent events to least, but you can click on the Table icon to view your results as a table, or you can click the Chart icon to view them as a chart. The Export button exports your search results in various formats: CSV, raw events, XML, or JSON.

SPL: Search Processing Language

Splunk helps sift data from the mass of indexed events into a form and search processing language has a various command that is useful for answering real-world questions.

The entire string

sourcetype=syslog ERROR | top user | fields - percent

is called a search, and the pipe character (|) separates the individual commands that make up the search.

Pipes

The first keyword after the pipe is the name of the search command. In this case, the commands are top and fields. What command is retrieving the events from the index? Well, there is an implied command called search, at the beginning of any search that doesn’t start with a pipe character. So, really, there are three search commands in the search: search, top, and fields.
The results from each command are passed as input to the next command.
If you have ever used a Linux shell such as bash.

Implied AND

sourcetype=syslog ERROR tells the search command to retrieve only events that have a sourcetype equal to Syslog AND contain the term ERROR.

Top user

The next command, top, returns the most common values of the specified fields. By default, top returns the top 10 most common values for the specified field, in descending order (thank you, David Letterman). In this case, the specified field is used, so the top returns the users that appear most often in Syslog events that contain the term ERROR. The output of the top is a table of 3 columns (user, count, and percent), with 10 rows of values.

It’s also important to understand that the output of the top command becomes the input to the next command after the pipe. In this sense, the top has transformed the search results into a smaller set of values.

Fields – percent

The second command, fields, with an argument of – percent, tells Splunk to remove the percent column from the output of the top command.

The search Command

The search command is the workhorse of Splunk. It’s one of the simplest and most powerful commands. It’s such a basic command that you don’t even need to type it anywhere before the first pipe, because it is invoked implicitly at the head of a search, retrieving events from the indexes on disk.

When it’s not the first command in a search, the search command can filter a set of results of the previous search. To do this, use the search command like any other command—with a pipe character followed by an explicit command name.

For example, the command error | top url
| search count>=2 searches for events on disk that have the word error, finds the top URLs and filters any URLs that only occur once. In other words, of the 10 error events that top returns, show me only the ones where there are two or more instances of that URL.

• Keyword arguments to the search command are not case-sensitive, but field names are.

• You need quotation marks around phrases or field values that contain breaking characters such as whitespace, commas, pipes, square brackets, and equals signs. So, host=web09 is fine, but if the host value has spaces, for example, you’ll need quotes around the value, as in host=”webserver #9”. In addition, to search for reserved keywords (e.g., AND, OR, NOT, etc.), use quotes.

• To search for quotes use a backslash to escape the quote character. To find the phrase—Splunk changed “life itself” for me—you’d search for: “Splunk changed ”life itself” for me”

• Arguments—keywords and fields—to the search command are ANDed together, implicitly.

• You can specify that either one of two or more arguments should be true using the OR keyword, in uppercase.

• To filter out events that contain a particular word, use the NOT keyword.

Subsearches

The search command, like all commands, can be used as a subsearch—a search whose results are used as an argument to another search command. Subsearches are enclosed in square brackets.

For example, to find all Syslog events from the user that had the last login error, use the following command:
sourcetype=syslog [search login error | return user]
Here, a search for events having the terms login and error is performed, returning the first user value found, say bob, followed by a search for sourcetype=syslog user=bob.