Searching with Splunk
The goal of search is to help you find exactly what you need. It can mean filtering, summarizing, and visualizing a large amount of data, to answer your questions about the data.
The Summary dashboard gives you a quick overview of the data visible to you. Click Launch search app on the Splunk Welcome tab. If you’re on the Splunk Home tab, click Search under Your Apps.
Few points about this dashboard:
- The search bar at the top is empty, ready for you to type in a search.
- The time range picker to the right of the search bar permits time range adjustment. You can see events from the last 15 minutes, for example, or any desired time interval. For real-time streaming data, you can select an interval to view, ranging from 30 seconds to an hour.
- The All indexed data panel displays a running total of the indexed data.
The next three panels show the most recent or common values that have been indexed in each category:
- The Sources panel shows which files (or other sources) your data came from.
- The Source types panel shows the types of sources in your data.
- The Hosts panel shows which hosts your data came from.
Search navigation menus near the top of the page includes:-
- Summary is where we are.
- Search leads to the main search interface, the Search dashboard.
- Status lists dashboards on the status of your Splunk instance.
- Dashboards & Views lists your dashboards and views.
- Searches & Reports lists your saved searches and reports.
3.1 The Search Dashboard
If you click the Search option or enter a search in the search bar, the page switches to the Search dashboard (sometimes called the timeline or flashtimeline view). When a search is kicked off, the results almost immediately start displaying. For example, entering an asterisk (*) in the search bar retrieves all the data in your default indexes.
The contents of this dashboard:-
- Timeline: A graphic representation of the number of events matching your search over time.
- Fields sidebar: Relevant fields along with event counts. This menu also allows you to add a field to the results.
- Field discovery switch: Turns automatic field discovery on or off. When Splunk executes a search and field discovery is on, Splunk attempts to identify fields automatically for the current search.
- Results area: Shows the events from your search. Events are ordered by Timestamp, which appears to the left of each event. Beneath the Raw text of each event are any fields selected from the Fields sidebar for which the event has a value.
When you start typing in the search bar, context-sensitive information appears below, with matching searches on the left and help on the right.
The search job controls are only active when a search is running. If you haven’t run a search, or if your search has finished, they are inactive and greyed out. But if you’re running a search that takes a long time to complete, you can use these icons to control the search progress:
- Sending a search to the background lets it keep running to completion on the server while you run other searches or even close the window and log out. When you click Send to background, the search bar clears and you can continue with other tasks. When the job is done, a notification appears on your screen if you’re still logged in; otherwise, Splunk emails you (if you’ve specified an email address). If you want to check on the job in the meantime, or at a later time, click the Jobs link at the top of the page.
- Pausing a search temporarily stops it and lets you explore the results to that point. While the search is paused, the icon changes to a play button. Clicking that button resumes the search from the point where you paused it.
- Finalizing a search stops it before it completes, but retains the results to that point and so you can view and explore it in the search view.
- In contrast, canceling a search stops it running, discards the results, and clears them from the screen.
The Job inspector icon takes you to the Job inspector page, which shows details about your search, such as the execution costs of your search, debug messages, and search job properties.
Use the Save menu to save the search, save the results, or save and share the results. If you save the search, you can find it on the Searches & Reports menu. If you save the results, you can review them by clicking on Jobs in the upper right corner of the screen.
Use the Create menu to create dashboards, alerts, reports, event types, and scheduled searches. Moving down to the upper left corner of the Results area, you see the following row of icons.
By default, Splunk shows events as a list, from most recent events to least, but you can click on the Table icon to view your results as a table, or you can click the Chart icon to view them as a chart. The Export button exports your search results in various formats: CSV, raw events, XML, or JSON.
3.2 SPL: Search Processing Language
Splunk helps sift data from the mass of indexed events into a form that is useful for answering real-world questions.
The entire string
sourcetype=syslog ERROR | top user | fields - percent
is called a search, and the pipe character (|) separates the individual commands that make up the search.
The first keyword after the pipe is the name of the search command. In this case the commands are top and fields. What command is retrieving the events from the index? Well, there is an implied command called search, at the beginning of any search that doesn’t start with a pipe character. So, really, there are three search commands in the search: search, top, and fields.
The results from each command are passed as input to the next command.
If you have ever used a Linux shell such as bash.
3.2.2 Implied AND
sourcetype=syslog ERROR tells the search command to retrieve only events that have a sourcetype equal to syslog AND contain the term ERROR.
3.2.3 Top user
The next command, top, returns the most common values of the specified fields. By default, top returns the top 10 most common values for the specified field, in descending order (thank you, David Letterman). In this case, the specified field is user, so top returns the users that appear most often in syslog events that contain the term ERROR. The output of top is a table of 3 columns (user, count, and percent), with 10 rows of values.
It’s also important to understand that the output of the top command becomes the input to the next command after the pipe. In this sense, top has transformed the search results to a smaller set of values.
3.2.4 Fields – percent
The second command, fields, with an argument of – percent, tells Splunk to remove the percent column from the output of the top command.
3.3 The search Command
The search command is the workhorse of Splunk. It’s one of the simplest and most powerful commands. It’s such a basic command that you don’t even need to type it anywhere before the first pipe, because it is invoked implicitly at the head of a search, retrieving events from the indexes on disk.
When it’s not the first command in a search, the search command can filter a set of results of the previous search. To do this, use the search command like any other command—with a pipe character followed by an explicit command name. For example, the command error | top url
| search count>=2 searches for events on disk that have the word error, finds the top URLs, and filters any URLs that only occur once. In other words, of the 10 error events that top returns, show me only the ones where there are two or more instances of that URL.
- Keyword arguments to the search command are not case-sensitive, but field names are.
- You need quotation marks around phrases or field values that contain breaking characters such as whitespace, commas, pipes, square brackets, and equals signs. So, host=web09 is fine, but if the host value has spaces, for example, you’ll need quotes around the value, as in host=”webserver #9”. In addition, to search for reserved keywords (e.g., AND, OR, NOT, etc.), use quotes.
- To search for quotes use a backslash to escape the quote character. To find the phrase—Splunk changed “life itself” for me—you’d search for: “Splunk changed \”life itself\” for me”
- Arguments—keywords and fields—to the search command are ANDed together, implicitly.
- You can specify that either one of two or more arguments should be true using the OR keyword, in uppercase.
- To filter out events that contain a particular word, use the NOT keyword.
The search command, like all commands, can be used as a subsearch—a search whose results are used as an argument to another search command. Subsearches are enclosed in square brackets. For example, to find all syslog events from the user that had the last login error, use the following command:
sourcetype=syslog [search login error | return user]
Here, a search for events having the terms login and error is performed, returning the first user value found, say bob, followed by a search for sourcetype=syslog user=bob.
Learn more about Splunk in this insightful blog now!