Spark SQL is mostly about distributed in-memory computations on massive scale.
Spark SQL provides Spark programmers with the benefits of relational processing (e.g., declarative queries and optimized storage), and allows SQL users to call complex analytics libraries in Spark (e.g., machine learning).
Now talking about index:
Indexing is done in order to allow you to move your disk head to the precise location on disk you want and just read that record.
Indexing input data
The reason why indexing over external data sources is not supported by Spark is that Spark is not a data management system but a batch data processing engine. Since the data used by spark comes from different sources and is not owned by spark, it cannot reliably monitor changes. And as a consequence it cannot maintain indices.
Now, if the data source that provides data to the Spark supports indexing, it will be indirectly utilized by Spark through mechanisms like predicate pushdown.
Indexing Distributed Data Structures:
standard indexing techniques require persistent and well defined data distribution but the exact distribution of the data present in Spark is nondeterministic and also this data is ephemeral.
high-level data layout achieved by proper partitioning plus attached columnar storage and compression provides an efficient distributed access without an overhead of creating, storing and maintaining indices. This common pattern is used by different in-memory columnar systems.
That being said some forms of indexed structures do exist in the Spark ecosystem.
Other projects, like Succinct (mostly inactive today) runs on a different approach and use advanced compression techniques with random access support.
And of course this raises a question. If there comes a case where we require an efficient random access why not use a system which is designed as a database from the beginning. There are many choices out there, including at least a few maintained by the Apache Foundation. At the same time Spark is evolving as a project. The future of Spark might hold anything.