LLM’s are very useful in research and analysis. In this blog I use Google NotebookLM to analyse DTCC CPMI-IOSCO QD documents to get an understanding of the benefits and pitfalls of using LLMs.

Background

Large Language Models (LLMs) are very useful in summarising and analysing data in documents. By using LLMs, we can improve our own productivity. Following on from last week’s blog, Using an LLM with Public CPMI-IOSCO Quantitative Disclosures for CCPs, in which I used Anthropic Claude, this week I will try Google NotebookLM for the same tasks.

CPMI-IOSCO Quantitative Disclosures (QDs)

On the Clarus Blog, I have written more than 25 articles on CCP Disclosures, one almost every quarter starting in 4Q 2016. This dataset is published quarterly by many Clearing Houses / Central Counterparties (CCPs) and contains over two hundred quantitative data fields covering margin, default resources, credit risk, collateral, liquidity risk, back-testing and more, with associated notes.

DTCC QDs for 3Q 2024

I will take the most recent DTCC QDs for Q3 2024 and compare these to the prior quarter’s publication. These are available in either xlsx or pdf form at the DTCC website. I downloaded the two pdfs and uploaded into Google NotebookLM, which is included with my Google Account subscription.

Let’s get started with Initial Margin

I entered the following prompt:

What are the values for disclosure reference 6.1.1.4 Total for each of GSD, MBSD and NSCC for both quarters for which there is data in the attached documents

Which returned:

So the same correct values as I got in last weeks blog from Claude LLM. As NotebookLM does not output visual charts, I used a slightly different prompt from last time. What is nice is that the references 1, 2, 3 & 4 in the output can be clicked upon and take you to the source text that has been read in from the pdfs.

So far so good, lets try something more difficult.

What has increased the most?

For this question, after a few trials, I entered the following prompt:

In the most recent quarter find those disclosure references only for GSD that have increased the most from their value at the prior quarter end and show just the top 5 in a visual table with rows ordered by highest percentage increase first

This returned output in 3 distinct sections.

A nice pre-amble explaining the process and highlighting the fact that the first part is Data Collection from the source document.

Then the table itself.

Unfortunately, instead of being the same as last week’s table produced by Claude, there is only one common row in the two tables!

That is not good.

Only 6.8.1 Maximum aggregate initial margin with a 55% increase is present in this table and the results from last week. Let’s also show the first table (before corrections) from the Claude blog and try and figure out what is going on.

Looking at the two source pdf documents and trying to explain myself:

• The Disclosure Reference “Special Charge” in the Claude results table is missing in the NotebookLM table, which is understandable as it is a supplementary disclosure without an official reference number and indeed got dropped by Claude from the third iteration of its table. So not too bad here.
• The Disclosure Reference 6.5.4 has indeed increased by 195% as identified by NotebookLM, but has been entirely missed by Claude! Presumably due to the code to parse the pdf document structure into tokens before feeding to the LLM. Looking at the source table, shown below, I can see it needs some ‘human like’“ intelligence to understand the data as the row for 6.5.4 and 6.5.4 is combined into one. However, visually as a human, I can see in the table which values relate to which disclosure:

• Disclosure Reference 6.8.1 is in both tables, hooray!
• Disclosure Reference 4.4.6.1 is not in the NotebookLM table but is in the Claude one and this time also caused by the same sort of ambiguity with two data elements in one row as thrown out NotebookLM and not Claude, a switcheroo from the prior bullet…
• Disclosure 4.3.7.1 – exactly the same situation as the previous bullet
• I will stop there….

So not great from either of our LLMs.

Accurate parsing the source document into data structures, if that is indeed what is going wrong here, is not a simple task when there is some ambiguity in the data formatting.

Generally data presented in csv or Excel format would be cleaner and most likely not have the above issues. Presumably it is not the LLM AI part going wrong here, but the code to parse the documents, While it is helpful that from my simple prompt the LLMs can make sense of the request and try and fulfil it, it is problematic that we cannot rely on the results being correct.

More Quarters of Data

One of the issues I had with Claude last week, was not being able to load five quarters of DTCC pdf files, even though each was 1.1 Mb in size and on the face of it should have worked.

Let’s try that with NotebookLM and ask it a simple question.

I tried the prompt:

What are the values for disclosure reference 4.1.4 for GSD, for all 5 quarters for which there is data in the attached documents, show the results in a visual table.

Unfortunately it took 3 iterations (!) with me having to ask NotebookLM to “please check these results as they look wrong to me for Q1 2024 and earlier quarters” and then again after more bad data, I entered“None of these are correct, please parse the source document again being careful to extract the correct values for GSD”, before I got a correct response.

Note the blurb at the top, helpful but annoying.

Hmm, perhaps we really need to treat LLMs as “new junior assistants” and as a rule ask them to double check their results and then check them ourselves before we get comfortable with their strengths and failings.

Learnings

My learnings from today.

If our source documents contain data tables, we are relying on LLMs parsing this data correctly.

Any ambiguity in the machine readability of data, such as that present in these DTCC pdf source documents, can throw out the analysis, making it very wrong.

We need to get into the habit of always asking LLMs to check their results.

Treat them as “new joiners” until we become familiar with their work.

What’s Next

• I would have liked to try the DTCC Excel workbooks as the source documents instead of pdfs.
• Unfortunately neither Claude or NotebookLM support reading data directly from XLSX, so I would have to create many CSV files from the workbooks.
• Or move to source data for a CCP that does publish CSV files.
• A topic to explore in a future blog.
• Stay tuned for that.
• I hope you found this blog informative.