Installing Elasticsearch
Okay, so in this video what we're going to do is actually index our data. So at the moment we just have all of our paragraphs from Meditations by Marcus Aurelius. And to do this, we are going to be using the Elasticsearch document store. So of course, if we're using Elasticsearch, we first need to actually download and install it. So I'm just going to take you through those steps now.
And all we need to do is head on over to this website up here, elasticsearch.co, and you can see the address just there. Now I'm going to follow the instructions for Windows, but of course if you're on Linux or Mac, just follow through. It's very similar either way. So here we're going to install it on Windows using the MSI installer.
So just scroll down here and we can see we can download the package from this link. So download that and once you download it, just open it and we'll see this window pop up. So once you see this window pop up, we just go through with all of the default settings. So install a service and continue through. Obviously if you do need to change anything, change it, but for me there's nothing here that I want to modify. Notice here we have the http port and we're using 9200. We'll be using that later. We just continue through here, default settings, and then we click install and we just let that install.
Verifying the Elasticsearch Cluster
Okay, so now that we've installed Elasticsearch, we can go ahead and actually check that it's running. So to do that, we're going to import Python requests. And whenever we interact with Elasticsearch, it's either going to be through Haystack or it will be through the requests library, and we'll just interact with the Elasticsearch API.
So to check the health of our cluster, so essentially check that's actually up and running, all we need to do is send a get request to localhost, and if you remember earlier, we had—it was port 9200. Of course, if the port on yours was different, modify it. This is just the default value. And after this, we need to reach out to the cluster endpoint, and then we are checking the health, and then we'll just format that as a JSON.
So what you should see here is we have our cluster, which is elasticsearch. May have a different name if you modified it, but by default, it's elasticsearch. The status is yellow which basically just means we have one node up and running. You can have multiple nodes in Elasticsearch, and for your cluster health to be green, it will expect your shards of indexes to have a backup shard across different nodes. And obviously we can't do that if we only have one node, but it's completely fine for us because we're just in development. If you're in production, yes, you'd probably want it to have those backup shards. If none of that made any sense, don't worry about it. We really don't need to know any of that for what we're doing here.
Checking for Existing Indices
Now what we can also do is we can check if we have any indices already. Now if I take a look at mine, I will already have some indices set up, which I've just set up prior to recording this. And to check that, we go to localhost again, and this time we want to call the cat API, which is what we would call whenever we want to see data in a table, human-readable format rather than JSON. And what we're checking here are the indices and we'll just add text onto there so we can actually see that. And this is quite messy, so if we just print it instead, it'll look a bit cleaner. Okay, so you can see I have these two indices. You shouldn't, I don't think, have either of those. No, you won't have either of those, so don't worry about that.
Now, what we are going to do is create a new index which will be called aurelius, and that is where we will put our documents.
Connecting Haystack to Elasticsearch
Now to actually implement that, we will be going through the Haystack library, which you can pip install farm-haystack. And what we want to do is from haystack.document_store.elasticsearch import ElasticsearchDocumentStore. So this is our document store instance, and of course, this is not aware of our Elasticsearch instance. We need to initialize that. So we'll store it in a variable called doc_store and all we write is ElasticsearchDocumentStore.
Now we need to initialize it with the parameters so it knows where to connect to our Elasticsearch instance. So to do that, we write host, and this is localhost. Now, if you have a username and password set, which you don't by default, you will need to enter them in here. I don't have any set, so no worries. And then we also need to specify our index. And at the moment we don't have an aurelius index, and that's fine because this will initialize it for us. So we'll just call it aurelius.
And if we go down here we can see what it actually did. So, it sent a PUT request to here: localhost:9200/aurelius. So that's how you create a new index.
Loading and Processing the Raw Text Data
After that, what we want to do is first import our data. So we have the data here, which I got from this website and processed with this script, which you can find on GitHub. I'll keep a link in the description so you can just go and copy that if you need to.
Now, I haven't really done much preprocessing; it's pretty straightforward. And all you need to do here is actually open that data. So we do that with open. And from here, that data file is located two folders up in a data folder. It's called meditations.txt. I'm going to be reading that, and all we do is data = f.read().
And then if we just have a quick look at the first 100 characters there, we see that we have this newline character, and that signifies a new paragraph from the text. So what we want to do here is split the data by newline. And then if we check the length of that, you see that we have 508 separate paragraphs in there.
So what we now want to do is we want to modify this data so that it's in the correct format for Haystack and Elasticsearch. So that format looks like this. It expects a list of dictionaries where each dictionary looks like this: the text, and inside here we would have our paragraph. So each one of these items here.
And then there's another optional field called meta. And meta contains a dictionary, and in here we can put whatever we want. So for us, I don't think at the moment there's really that much to put into here other than where it came from. So the the book, or maybe maybe the source is probably a better word to use here. And all of these are coming from Meditations. Now later on, we will probably add a few other books as well, and then the source will be different. And when we return that item from our retriever and our reader, we'll at least be able to see which book it came from. It would be also be pretty cool to maybe include like a page number or something, but at the moment with this, there are no page numbers included, so we don't—we're not doing that at the moment.
So that's the format that we need and it's going to be a list of these. So to do that, we'll just do some list comprehension. So we're going to write this, and let's just copy this. I think, yeah, it should be fine. We'll copy this and just indent that. And in here we have our paragraph, and sources Meditations for all of them. And then we just write for paragraph in data. Okay, so yeah, that should work. And if we just check what we have here. Okay so that's what we want. So we have text, we have a paragraph and then in here we have this meta with a source, which is always Meditations at the moment. So that looks pretty good. And we'll just double-check the length again. It should be 508. Okay, perfect.
Indexing Documents with Haystack
Now what we need to do is index all of these documents into our Elasticsearch instance. And to do that, it's super easy. All we do is call doc_store because we're doing this through Haystack now, and we do write_documents, and we just pass in our data.json. And that should work. Okay, cool. So we can see here what it's done as it's sent a post request to the bulk API. And sent two of them, I assume because it can only send so many documents at once. So that's pretty cool.
And now what I want to check is that we actually have 508 documents in our Elasticsearch instance. So to do that, we're going to revert back to requests. So we'll do requests.get. Again, go to our localhost 9200, and here we need to specify the index that we want to count the number of entries in. And then all we do is add _count on to the end there. And this will return a JSON object, so we do this so that we can see it. And sure enough, we have 508 items in that document store.
Conclusion and What's Next
So if we head on back to our original plan, so up here we had Meditations. We've now got that and we've also set up the first part of our stack over here. So Elastic now has Meditations in there, so we can cross that off. Now the next step is setting up our retriever, which we'll cover in the next video.
So that's everything for this video. I hope you enjoyed, and I will see you again in the next one.
