Splunk Interview Questions and Answers

CTA

Splunk is among the best load management and analysis solutions in the IT industry. It is one of the top analytics and Big Data tools; it has an extremely high demand in the corporate world, and so is the case with Splunk professionals. If you wish to become a successful Big Data engineer, having expert knowledge and skills in Splunk is important. This is why the competition in the market is challenging for Splunk jobs. Here, we have created a list of Splunk interview questions and answers with the help of industry experts so that you can be prepared for your interview. Take a look at the following Splunk interview questions and get ready to crack your interview:

The Splunk Interview Questions blog is largely divided into the following parts:

1. Basic interview questions

2. Intermediate interview questions

3. Advanced interview questions

4. Splunk Admin Interview Questions

Check this video on Splunk software engineer interview questions and answers:

1. Compare Splunk with Spark.

2. What is Splunk?

Splunk is ‘Google’ for our machine-generated data. It’s a software/engine that can be used for searching, visualizing, monitoring, reporting, etc. of our enterprise data. Splunk takes valuable machine data and turns it into powerful operational intelligence by providing real-time insights into our data through charts, alerts, reports, etc.

3. What are the common port numbers used by Splunk?

Below are the common port numbers used by Splunk. However, we can change them if required.

4. What are the components of Splunk? Explain Splunk architecture.

This is one of the most frequently asked Splunk interview questions. Below are the components of Splunk:

• Search Head: Provides the GUI for searching
• Indexer: Indexes the machine data
• Forwarder: Forwards logs to the Indexer.
• Deployment Server: Manages Splunk components in a distributed environment.

5. Which is the latest Splunk version in use?

Splunk 8.2.1 (as of June 21, 2021)

6. What is a Splunk indexer? What are the stages of Splunk indexing?

A Splunk indexer is the Splunk Enterprise component that creates and manages indexes. The primary functions of an indexer are mentioned below:

• Indexing incoming data
• Searching the indexed data
• Picture

7. What is a Splunk forwarder? What are the types of Splunk forwarders?

There are two types of Splunk forwarders, which are mentioned below:

• Universal Forwarder (UF): the Splunk agent installed on a non-Splunk system to gather data locally; it can’t parse or index data.
• Heavyweight Forwarder (HWF): A full instance of Splunk with advanced functionalities.

It generally works as a remote collector, intermediate forwarder, and possible data filter, and since it parses data, it is not recommended for production systems.

8. Can you name a few most important configuration files in Splunk?

• props.conf
• indexes.conf
• inputs.conf
• transforms.conf
• server.conf

9. What are the types of Splunk Licenses?

• Enterprise license
• Free license
• Forwarder license
• Beta license
• Licenses for search heads (for distributed search)
• Licenses for cluster members (for index replication)

CTA

10. What is the Splunk app?

The Splunk app is a container or directory of configurations, searches, dashboards, etc. in Splunk.

11. Where is the Splunk default configuration stored?

$splunkhome/etc/system/default

12. What are the features not available in Splunk Free?

Splunk Free does not include below features:

• Authentication and scheduled searches/alerting
• Distributed search
• Forwarding in TCP/HTTP (to non-Splunk)
• Deployment management

13. What happens if the license master is unreachable?

If the license master is not available, the license slave will start a 24-hour timer, after which the search will be blocked on the license slave (though indexing continues). However, users will not be able to search for data in that slave until it can reach the license master again.

14. What is a summary index in Splunk?

A summary index is the default Splunk index (the index that Splunk Enterprise uses if we do not indicate another one).

If we plan to run a variety of summary index reports, we may need to create additional summary indexes.

15. What is Splunk DB Connect?

Splunk DB Connect is a generic SQL database plugin for Splunk that allows us to easily integrate database information with Splunk queries and reports.

16. Can you write down a general regular expression for extracting the IP address from logs?

There are multiple ways in which we can extract the IP address from logs. Below are a few examples:

By using a regular expression:

rex field=_raw  "(?<ip_address>d+.d+.d+.d+)"

OR

rex field=_raw  "(?<ip_address>([0-9]{1,3}[.]){3}[0-9]{1,3})"

17. Explain Stats vs Transaction commands.

This is another frequently asked interview question on Splunk that will test the developer’s or engineer’s knowledge. The transaction command is most useful in the following two specific cases:

• When the unique ID (from one or more fields) alone is not sufficient to discriminate between two transactions. This is the case when the identifier is reused, for example, in web sessions identified by a cookie or client IP. In this case, the time span or pauses are also used to segment the data into transactions.
• When an identifier is reused, say, in DHCP logs, a particular message identifies the beginning or end of a transaction.
• When it is desirable to see the raw text of events combined rather than an analysis of the constituent fields of the events.

In other cases, it’s usually better to use stats.

• As the performance of the stats command is higher, it can be used, especially in a distributed search environment.

If there is a unique ID, the stats command can be used

18. How do I troubleshoot Splunk performance issues?

The answer to this question would be very wide, but, mostly, an interviewer would be looking for the following keywords:

• Check splunkd.log for errors
• Check server performance issues, i.e., CPU, memory usage, disk I/O, etc.
• Install the SOS (Splunk on Splunk) app and check for warnings and errors in its dashboard
• Check the number of saved searches currently running and their consumption of system resources
• Install and enable Firebug, which is a Firefox extension. Log into Splunk (using Firefox) and open Firebug’s panels. Then, switch to the ‘Net’ panel, which we will have to enable. The Net panel will show us the HTTP requests and responses, along with the time spent on each. This will give us a lot of information quickly, such as which requests are hanging Splunk, which requests are blameless, etc.

19. What are buckets? Explain the Splunk bucket lifecycle.

Splunk places indexed data in directories, which are called ‘buckets.’ It is physically a directory containing events from a certain period.

A bucket moves through several stages as it ages. Below are the various stages it goes through:

• Hot: A hot bucket contains newly indexed data. It is open for writing. There can be one or more hot buckets for each index.
• Warm: A warm bucket consists of data rolled out from a hot bucket. There are many warm buckets.
• Cold: A cold bucket has data that is rolled out from a warm bucket. There are many cold buckets.
• Frozen: A frozen bucket is comprised of data rolled out from a cold bucket. The indexer deletes frozen data by default, but we can archive it. Archived data can later be thawed (data in a frozen bucket is not searchable).

By default, the buckets are located in the following location:

$SPLUNK_HOME/var/lib/splunk/defaultdb/db

We should see the hot-db there and any warm buckets we have. By default, Splunk sets the bucket size to 10 GB for 64-bit systems and 750 MB for 32-bit systems.

20. What is the difference between stats and eventstats commands?

• The stats command generates summary statistics of all the existing fields in the search results and saves them as values in new fields.
• Eventstats is similar to the stats command, except that the aggregation results are added inline to each event and only if the aggregation is pertinent to that event. The eventstats command computes requested statistics, much like how stats do, but aggregates them to the original raw data.

21. Who are the top direct competitors to Splunk?

Logstash, Loggly, LogLogic, Sumo Logic, etc. are some of the top direct competitors to Splunk.

22. What do Splunk licenses specify?

Splunk licenses specify how much data we can index per calendar day.

23. How does Splunk determine 1 day, from a licensing perspective?

In terms of licensing, for Splunk, one day is from midnight to midnight on the clock of the license master.

24. How are forwarder licenses purchased?

They are included in Splunk. Therefore, there is no need to purchase them separately.

25. What is the command for restarting Splunk web server?

This is another frequently asked Splunk commands interview question. Get a thorough idea of commands We can restart the Splunk web server by using the following command:

splunk start splunkweb

26. What is the command for restarting the Splunk Daemon?

Splunk Deamon can be restarted with the below command:

splunk start splunkd

27. What is the command used to check the running Splunk processes on Unix/Linux?

If we want to check the running Splunk Enterprise processes on Unix/Linux, we can make use of the following command:

ps aux | grep splunk

28. What is the command used for enabling Splunk to boot start?

To boot start Splunk, we have to use the following command:

$SPLUNK_HOME/bin/splunk enable boot-start

29. How to disable Splunk boot-start?

In order to disable Splunk boot-start, we can use the following:

$SPLUNK_HOME/bin/splunk disable boot-start

30. What is a source type in Splunk?

The source type is Splunk way of identifying data.

31. How to reset the Splunk admin password?

Resetting the Splunk admin password depends on the version of Splunk. If we are using Splunk 7.1 and above, then we have to follow the below steps:

• First, we have to stop our Splunk Enterprise
• Now, we need to find the ‘passwd’ file and rename it to ‘passwd.bk’
• Then, we have to create a file named ‘user-seed.conf’ in the below directory:

$SPLUNK_HOME/etc/system/local/

In the file, we will have to use the following command (here, in place of ‘NEW_PASSWORD’, we will add our own new password):

[user_info]

PASSWORD = NEW_PASSWORD

• After that, we can just restart the Splunk Enterprise and use the new password to log in

Now, if we are using versions prior to 7.1, we will follow the below steps:

• First, stop the Splunk Enterprise
• Find the passwd file and rename it to ‘passw.bk.’
• Start Splunk Enterprise and log in using the default credentials of admin/changeme.
• Here, when asked to enter a new password for our admin account, we will follow the instructions

Note: In case we have created other users earlier and know their login details, copy and paste their credentials from the passwd.bk file into the passwd file and restart Splunk.

32. How to disable the Splunk launch message?

Set value OFFENSIVE=Less in splunk_launch.conf

Learn more from Intellipaat’s insightful Splunk Tutorial!

33. How to clear the Splunk search history?

We can clear the Splunk search history by deleting the following file from the Splunk server:

$splunk_home/var/log/splunk/searches.log

34. What is Btool? How will you troubleshoot Splunk configuration files?

Splunk Btool is a command-line tool that helps us troubleshoot configuration file issues or just see what values are being used by our Splunk Enterprise installation in the existing environment.

35. What is the difference between the Splunk app and Splunk add-ons?

In fact, both contain preconfigured configuration, reports, etc., but the Splunk add-on does not have a visual app. On the other hand, a Splunk app has a preconfigured visual app.

36. What is the ‘.conf’ file’s precedence in Splunk?

File precedence is as follows:

System local directory — highest priority

App local directories

App default directories

System default directory — lowest priority

37. What is a fishbucket? What is a fishbucket index?

Fishbucket is a directory or index at the default location:

/opt/splunk/var/lib/splunk

It contains seek pointers and CRCs for the files we are indexing, so ‘splunkd’ can tell us if it has read them already. We can access it through the GUI by searching for:

index=_thefishbucket

38. How do I exclude some events from being indexed by Splunk?

This can be done by defining a regex to match the necessary event(s) and sending everything else to NullQueue. Here is a basic example that will drop everything except events that contain the string login:
In props.conf:

[source::/var/log/foo]

# Transforms must be applied in this order

# to make sure events are dropped on the

# floor prior to making their way to the

# index processor

TRANSFORMS-set= setnull,setparsing

In transforms.conf:

[setnull] REGEX = . DEST_KEY = queue FORMAT = nullQueue

[setparsing]

REGEX = login

DEST_KEY = queue

FORMAT = indexQueue

39. How can I understand when Splunk has finished indexing a log file?

We can figure this out:
By watching data from Splunk’s metrics log in real-time:

index="_internal" source="*metrics.log" group="per_sourcetype_thruput" series="<your_sourcetype_here>" |eval MB=kb/1024 | chart sum(MB)

By watching everything split by source type:

index="_internal" source="*metrics.log" group="per_sourcetype_thruput" | eval MB=kb/1024 | chart sum(MB) avg(eps) over series

If we are having trouble with data input and we want a way to troubleshoot it, particularly if our whitelist/blacklist rules are not working the way we expected, we will go to the following URL:

https://yoursplunkhost:8089/services/admin/inputstatus

40. How to set the default search time in Splunk 6?

To do this in Splunk Enterprise 6.0, we have to use ‘ui-prefs.conf’. If we set the value in the following, all our users would see it as the default setting:

$SPLUNK_HOME/etc/system/local

For example, if our

$SPLUNK_HOME/etc/system/local/ui-prefs.conf file

includes:

[search]
dispatch.earliest_time = @d
dispatch.latest_time = now

The default time range that all users will see in the search app will be today.

The configuration file reference for ui-prefs.conf is here:

http://docs.splunk.com/Documentation/Splunk/latest/Admin/Ui-prefsconf

41. What is a dispatch directory?

$SPLUNK_HOME/var/run/splunk/dispatch

contains a directory for each search that is running or has completed. For example, a directory named 1434308943.358 will contain a CSV file of its search results, a search.log with details about the search execution, and other stuff. Using the defaults (which we can override in limits.conf), these directories will be deleted 10 minutes after the search completes—unless the user saves the search results, in which case the results will be deleted after 7 days.

42. What is the difference between search head pooling and search head clustering?

Both are features provided by Splunk for the high availability of Splunk search head in case any search head goes down. However, the search head cluster feature has only recently been introduced, while the search head pooling feature will be removed in the next few versions.

The search head cluster is managed by a captain, and the captain controls its slaves. The search head cluster is more reliable and efficient than the search head pooling.

43. If I want to add folder access logs from a windows machine to Splunk, how do I do it?

Below are the steps to add folder access logs to Splunk:

1. Enable Object Access Audit through group policy on the Windows machine on which the folder is located
2. Enable auditing on a specific folder for which we want to monitor logs
3. Install the Splunk universal forwarder on the Windows machine.
4. Configure the universal forwarder to send security logs to the Splunk indexer.

44. How would you handle/troubleshoot a Splunk license violation warning?

A license violation warning implies that Splunk has indexed more data than our purchased license quota. We have to identify which index/source type has received more data recently than the usual daily data volume. We can check the Splunk license master pool-wise available quota and identify the pool in which the violation has occurred. Once we identify the pool that is receiving more data, we have to identify the top source type that is receiving more data than usual. Once the source type is also identified, we find the source machine that is sending the huge number of logs and, in turn, the root cause for the same, and troubleshoot it accordingly.

Find out how Splunk’s machine learning can transform your business operations, and help you make better, data-driven decisions.

45. What is MapReduce algorithm?

MapReduce algorithm is the secret behind Splunk’s faster data searching. It’s an algorithm typically used for batch-based large-scale parallelization. It’s inspired by functional programming’s map() and reduce() functions.

46. How does Splunk avoid the duplicate indexing of logs?

At the indexer, Splunk keeps track of the indexed events in a directory called Fishbucket with the following default location:

/opt/splunk/var/lib/splunk

It contains seek pointers and CRCs for the files we are indexing, so splunkd can tell us if it has read them already.

See more at:

http://www.learnsplunk.com/splunk-indexer-configuration.html#sthash.t1ixi19P.dpuf.

CTA

47. What is the difference between the Splunk SDK and the Splunk Framework?

Splunk SDKs are designed to allow us to develop applications from scratch; they do not require Splunk Web or any components from the Splunk App Framework. These are separately licensed from Splunk, and they do not alter the Splunk software.

Splunk App Framework resides within the Splunk web server and permits us to customize the Splunk Web UI that comes with the product and develop Splunk apps using the Splunk web server. It is an important part of the features and functionalities of Splunk, which does not license users to modify anything in Splunk.

48. For what purpose inputlookup and outputlookup are used in Splunk Search?

The inputlookup command is used to search the contents of a Splunk lookup table. The lookup table can be a CSV lookup or a KV store lookup. The inputlookup command is considered an event-generating command. An event-generating command generates events or reports from one or more indexes without transforming them. There are numerous commands that come under event-generating commands, including metadata, loadjob, inputcsv, etc. The inputlookup command is event-generating.

Syntax:

inputlookup [append=] [start=] [max=] [ | ] [WHERE ]

Now coming to the outputlookup command, it writes the search results to a static lookup table, or KV store collection, that we specify. The outputlookup command is not being used with external lookups.

Syntax:

outputlookup [append=<bool>] [create_empty=<bool>] [max=<int>] [key_field=<field_name>] [createinapp=<bool>] [override_if_empty=<bool>] (<filename> | <tablename>)

49. Explain how Splunk works.

We can divide the working of Splunk into three main parts:

• Forwarder: You can see it as a dumb agent whose main task is to collect the data from various sources like remote machines and transfer it to the indexer.
• Indexer: The indexer processes the data in real time and stores and indexes it on the localhost or cloud server.
• Search Head: It allows the end-user to interact with the data and perform various operations like searching, analyzing, and visualizing the information.

50. How to add the colors in Splunk UI based on the field names?

Splunk UI has a number of features that allow the administrator to make the reports more presentable. One such feature that proves to be very useful for presenting distinguished results is the custom colors. For example, if the sales of a product drop below a threshold value, then as an administrator you can set the chart to display the values in red color.

The administrator can also change chart colors in the Splunk Web UI by editing the panels from the panel settings mentioned above the dashboard. Moreover, you can write the codes and use hexadecimal values to choose a color from the palette.

51. How the Data Ages in Splunk?

The data that is entering an indexer gets sorted into directories, which are also known as buckets. Over a period of time, these buckets roll over different stages, from hot to warm, cold to frozen, and finally thawed. The indexer goes through a pipeline where event processing takes place. It occurs in two stages: parsing breaks them into individual events, while indexing takes these events into the pipeline for processing.

This is what happens to the data at each stage of the indexing pipeline:

• As soon as the data center the pipeline, it goes to the hot bucket. There can be multiple hot buckets at any point in time, which you can both search and write to.
• If any problem like the Splunk getting restarted or the hot bucket has reached a certain threshold value/size, then a new bucket will be created in its place and the existing ones roll to become a warm bucket. These warm buckets are searchable, but you cannot write anything in them.
• Further, if the indexer reaches its maximum capacity, the warm bucket will be rolled to become a cold one. Splunk will automatically execute the process by selecting the oldest warm bucket from the pipeline. However, it doesn’t rename the bucket. All the above buckets will be stored in the default location ‘$SPLUNK_HOME/var/lib/splunk/defaultdb/db/*’.
• After a certain period of time, the cold bucket rolls to become the frozen bucket. These buckets don’t have the same location as the previous buckets and are non-searchable. These buckets can either be archived or deleted based on the priorities.
• You can’t do anything if the bucket is deleted, but you can retrieve the frozen bucket if it’s being archived. The process of retrieving an archived bucket is known as thawing. Once a bucket is thawed it becomes searchable and stores into a new location

  ‘$SPLUNK_HOME/var/lib/splunk/defaultdb/thaweddb/’
  

52. What are pivots and data models in Splunk?

Data models in Splunk are used when you have to process huge amounts of unstructured data and create a hierarchical model without executing complex search queries on the data. Data models are widely used for creating sales reports, adding access levels, and creating a structure of authentication for various applications.

Pivots, on the other hand, give you the flexibility to create multiple views and see the results as per the requirements. With pivots, even the managers of stakeholders from non-technical backgrounds can create views and get more details about their departments.

53. Explain workflow actions.

This topic will be present in any set of Splunk interview questions and answers. Workflow actions in Splunk are referred to as highly configurable, knowledge objects that enable you to interact with web resources and other fields. Splunk workflow actions can be used to create HTML links and use them to search field values, put HTTP post requests for specific URLs, and run secondary searches for selected events.

54. How many types of dashboards are available in Splunk?

There are three types of dashboards available in Splunk:

• Real-time dashboards
• Dynamic form-based dashboards
• Dashboards for scheduled reports

55. What are the types of alerts available in Splunk?

Alerts are the actions generated by a saved search result after a certain period of time. Once an alert has occurred, subsequent actions like sending an email or a message will also be triggered. There are two types of alters available in Splunk, which are mentioned below:

Types of alters available in Splunk:

• Real-Time Alerts: We can divide the real-time alerts into two parts: pre-result and rolling-window alerts. The pre-result alert gets triggered with every search, while rolling-window alerts are triggered when a specific criterion is met by the search.
• Scheduled Alerts: As the name suggests, scheduled alerts can be initialized to trigger multiple alerts based on the set criteria.

56. Define the terms ‘search factor’ and ‘replication factor.’

Search factor: The search factor (SF) decides the number of searchable copies an indexer cluster can maintain of the data/bucket. For example, the search factor value of 3 shows that the cluster can maintain up to 3 copies of each bucket.

Replication factor: The replication factor (RF) determines the number of users that can receive copies of your data/buckets. However, the search factor should not be greater than the replication factor.

57. How to stop/start the Splunk service?

The command for starting Splunk service:

./splunk start

The command for stopping Splunk service:

./splunk stop

58. What is the use of a ‘time zone’ property in Splunk?

Time Zone is an important property that helps you search for the events in case any fraud or security issue occurs. The default time zone will be taken from the browser settings or the machine you are using. Apart from event searching, it is also used in data pouring from multiple sources and aligns them based on different time zones.

59. What are the important Search commands in Splunk?

Below are some of the important search commands in Splunk:

• Erex
• Abstract
• Typer
• Rename
• Anomalies
• Fill down
• Accum
• Add totals

60. How many types of search modes are there in Splunk?

There are three types of search modes in Splunk:

• Fast mode: speeds up your search result by limiting the types of data.
• Verbose mode: Slower as compared to the fast mode, but returns the information for as many events as possible.
• Smart mode: It toggles between different modes and search behaviors to provide maximum results in the shortest period of time.

CTA