Introduction to CharacterTextSplitter
Hello guys, in this video we are going to look at the first type of chunking method which is going to be using the character text splitter. And yes, we did look at it in the first few videos because this is going to give us a nice foundation to learn about more advanced concepts like recursive character text splitter and semantic chunking. All right, so let us see how it works.
So the character splitter doesn't just cut at character limits. It follows a split first, a merge second approach. So essentially, first off, it is going to break text at a separator. This is something that we have to specify, but by default it's going to be a double new line character. Let's look at the second one. The next step is going to be merging. It's going to combine the pieces until hitting the chunk size limit. So this is something that we do specify. If you don't understand this, don't worry. Let's look at a very small example.
So let's say this is going to be the example and this is going to be the configuration of our character text splitter. So we're essentially saying that the chunk size is going to be 100 characters and this is going to be the default separator. We can either mention it or not mention it. So if I come back to the code that we have, so if I go back to the injection pipeline coming all the way up. Okay. So right here, chunk size is something that we do specify. We're specifying here, you know, 800 characters. So for the sake of learning, we're just going to have it as 100 characters right here. And as a separator, if we actually go inside of here and look at the separator, you should be able to see that this is going to be the default value. It's a double new line.
So coming back to our example right here. So essentially what we see right here, the space right here between the lines, these are essentially going to be the double new line. If there was no space, if it was just in the next line, then it's going to be a single new line. But because there is a space right here, it's going to be a double new line. So let's try to understand what's going to happen. Let's come back to our sequence right here. So what does the character text do? It's going to first split it at whichever separator that we have mentioned right here. In this case, it's going to be a double new line. So what's going to happen? So, uh, let's call it pieces, right?
So this is going to be piece one. This is going to be piece two, piece three, piece four, piece five, piece six. So that is exactly what has happened right here. Okay. So at the end of the first step, it basically goes through the entire document. It's going to split it into pieces based on whatever separator that we have mentioned. That's going to be the step one. The step two, okay, step two, what's going to happen? Let's come back.
How CharacterTextSplitter Merges Chunks
So the step two is it's going to combine the pieces until hitting the chunk size limit. So what does it mean? So we know that the chunk size that we have configured, hypothetically, is going to be 100 characters. So if you come here, I've also, to make it very easier for you to understand, so we also have, you know, I've also mentioned the character counts for each of the pieces. So 18, 51, 19, 78, 21, 62.
Okay. So we know that the next step is going to be merging. It's going to combine the pieces until hitting the chunk size limit. So this is what's going to happen. Um, so it's going to look at piece one. It's going to see, okay, this is 18 characters. It is still less than 100 characters. So, it's going to look at piece 2. It's going to see this is 51 characters. So, it's it's going to try to combine uh it's going to add up 18 and 51 characters and then see, okay, 18 + 51 is still less than 100, right? So, it's still going to be less than 100 because that is going to be the chunk size. So because it is going to be less than 100, piece one and piece two is going to be merged into a single chunk. Okay, it's not going to stop there. It's now going to look at the next piece.
Okay, so I'm not really sure if 18 + 51, it's what is that, 59, 69, and 69 + 19 is still going to be under 100. So this entire thing is going to be part of the same chunk. And then again it's going to, you know, see if these three can be merged with the next one. So it's going to try to add up all these different values. And then it's going to see if I were to add this with 78, will I exceed the 100 character limit or will I be under? So if I'm going to be under, in that case, great, it's going to add it up. But here you can see that the next sentence is going to be 78 characters and definitely in that case it's going to go above this 100 characters. So what it's going to do is it is going to split. Okay, it's going to draw a line right here.
So this is going to be a part of one chunk and then the next chunk is going to start from here. Okay, the same thing is going to continue. 78 + 21. Okay, it's still 100. It it is still under 100. So it's going to try to add sentence one, piece 4 and piece five. And this is going to be a single chunk because if it were to add P6 in that case, it's it's going to go above 100 in that case. Okay, it's going to, you know, it's going to break the rule that we have set right here, which is going to be chunk size 100. Right, so this is how it works. I hope this makes sense.
The Major Drawback: Oversized Chunks
Okay, so you can see that, you know, after merging, the final chunk is going to be this one. You can see the double line character right here. It's going to, it basically added piece one, piece two, piece three and chunk two, you can see it only has the fourth piece, and chunk three has piece five and piece six. And if you look at the characters, you can see it is well within the 100 characters that we had set for a particular chunk. Although this looks pretty good, there is going to be one huge disadvantage when it comes to character text splitter. That is kind of what the next text splitter that we are going to be looking at, the recursive character text splitter, is going to solve.
So let's look at what is the drawback here. So the drawback is, let's say what happens when a single split exceeds the chunk size. So what I mean by that is, okay, so this is piece one, piece two, piece three, whatever, but what if there is a piece, that particular piece itself exceeds 100 characters? Okay, so that is going to be the scenario. So what happens if a single piece is larger than chunk size of 100? So let's say this is going to be the text. You can see that this text is going to be close to 200 characters, but we still have the configuration for the chunk size to be 100 characters. So what's going to happen? This cannot possibly be split further down because we have specified that the separator is going to be the double new line, but there is no double new line character right here. It's just one long sentence, right? So in this case, what's going to happen is the character splitter keeps this as is, even though the chunk exceeds the limit. Okay, so this is going to be the disadvantage.
So that is probably why during, you know, one of the initial videos. Okay, so in the injection pipeline. Okay, so let me actually go ahead and run this again so that I'd be able to show you. So if I come all the way up, you can actually see that during the chunking process, it says that it created a chunk of size somewhere around 900, which is longer than the specified 800. Right? So if I come up here, you can see that I have specified the chunk size to be 800, but there's a lot of chunks that are exceeding that particular value. So you know, 991, which is greater than 800; 1510, which is greater than 800. And the reason why is that if we go back to our docs, okay, let's go back to one of these docs. So if we look through here, there are going to be some paragraphs that are going to be much more bigger and we just cannot split it even further because there is no double new line right here. So in that case, it's going to keep this intact.
A Flawed Solution: Changing the Separator
So to combat this, okay, there's a couple of things we can do. So let's actually come to this character text splitter file. You can see this is going to be the paragraph that we are going to be working with. And each of these lines are going to be well within the 100 character limit. But this last line alone, you can see that, you know, this is one very long paragraph that definitely exceeds our 100 character limit and has no double new lines inside it whatsoever, making it impossible to split properly. So if I run this file right here, so you can see that we have chunk one, chunk two, chunk three, chunk four. So chunk 1, 2, 3, they're all satisfying the condition of being below 100 characters. But chunk four, despite this being, you know, 167 characters, it is still being intact right here because the splitter cannot split this even further. Okay. Because we've only specified the double new line.
If we want it to be able to split it even further, then we might have to change the separator value. So we know that the separator is going to be a double new line right here. But we will have to provide it something else. Okay, we can try to provide, you know, just slash n. In that case, it's just going to be a new line. But even then it's not going to work on this particular line because there is no second line. Everything is just in one line, right? It cannot split it even further. So if I go ahead and run this, it basically has the same thing right here. It doesn't have much difference. So what else can we provide here? So there's a couple of values that we can provide. So we've tried this one. Okay, this has the highest priority.
If that is not going to be possible, then we can try just another line. That is not going to work as well. And then we have the period, basically the period at the end of a sentence. Okay, here that is not going to work either because the period is at the very end. It cannot split it right here because after that there is no line there. So how can we split this line even further? So we can try, you know, the empty space character right here. Okay. So it can basically look at this entire thing and let's say this is going to... this is going to be the first 100 characters. So let's say this, till this "no" is going to be the first 100 characters. It's basically going to see, okay, we have hit our limit. We are going to chunk it right here so that this is going to be in another chunk. Okay, so we can try that. So let me actually show you what happens. So this is going to be an empty character. Let's clear it out and let's run it again.
Great. So in this case, you can actually see that, you know, it did chunk it. So if I come all the way up, you can see chunk one is 97 character, chunk two is 86, chunk three is 97, chunk four is 94, chunk five is 50. They're all under 100. Okay, so it did work. So it does appear to have worked. But if you take a closer look at what happened, it messed up a lot of other things. So you can see that Tesla reported record revenue of this thing. And if you see this is where the chunk one ends. So it ended somewhere in a very awkward place, right? So it ended right here after the. Okay. So apparently this probably would have been the, the 100 characters and it split right here. But this is not right. Correct. The chunk has to be sort of compact. It needs to have all the information. It cannot be borrowing some words from the next chunk and it cannot jumble things up. It should always follow this particular priority.
So first it should always give priority to double line. If that is not possible, only then it should go for a single line. If that is not possible, only then it should go for a dot. If that is not possible, only then it should go for an empty space. So for these particular lines there is no necessary to split it at empty space or a period or anything because we already have the double line. Only for this particular sentence do we have to resort to this particular separator.
Introducing the RecursiveCharacterTextSplitter
So this is going to be the drawback of the character text splitter. The only drawback that we saw is that we can only provide one separator right here. But wouldn't it be great if we could provide a list of these separators and basically say that prioritize this first. If that is not possible, prioritize this. So we are providing an order or a list of, you know, separators in the decreasing order of priority. If we could do that, wouldn't that be, wouldn't that solve this particular problem?
Well, that is exactly what the next separator that we are going to be looking at, the next type of chunking that we're going to be looking at, which is the RecursiveCharacterTextSplitter. So, if we've understood this, the second one is going to be super easy. So, all that this does, so let me just comment this out. So, this is going to be example two. Okay, so all that this does is that it by default takes in a list of separators and this is actually going to be the default value. Okay, so exactly what we saw right here in the character text splitter, we can only provide one separator at a given time. But in a recursive character text splitter, it's going to recursively, okay, it's going to look at a particular piece and recursively it is going to try to chunk it. It's going to further split it if it doesn't meet the chunk size. So that is what this is going to, it basically solves the drawback that we saw with the character text.
So that is it for the character text as well as the recursive character text splitter. In the next video we're going to be looking at an even more interesting, fun chunking method which is going to be the semantic chunking where we are actually going to be using embedding models to figure out, you know, exactly where to chunk the chunks based on the changes in the meaning between sentences. So that's going to be pretty interesting.
