Jump from previous section to creating reasoning dataset is bit unclear to me. Why are we doing this step? What is the value?

Good Q! I edited the README to make it clearer. Thanks @varunfb for all the great feedback.

BTW, i've done fine-tuning with the reasoning dataset and am running the eval script (each inference takes longer). Will update the README (and remove the 1st in Next Steps if needed) after the eval is done. Plan to merge the PR next week.

If we have done the FT with adding reasoning to the sql dataset, we should keep it but we need to better frame and provide more context why and how we have done it. I am guessing synthetic data was part of it. @jeffxtang

cc @init27 please review the narrative.

If we have done the FT with adding reasoning to the sql dataset, we should keep it but we need to better frame and provide more context why and how we have done it. I am guessing synthetic data was part of it. @jeffxtang

The why and how are stated here: "The goal is to see if we can improve the accuracy of the fine-tuned model by adding the reasoning info in the dataset." followed by Creating a reasoning dataset from the TRAIN dataset. The dataset was generated synthetically and I thought about using the synthetic-data-kit but felt like it's an overkill. But porting this script to using the data kit would be more than welcome! @init27

import { createClient } from 'redis';

const client = createClient();
await client.connect();

await client.json.set('user:1', '$', {
name: 'Alice',
emails: ['alice@example.com', 'alice@work.com'],
address: { city: 'NYC', zip: '10001' }
});

console.log(await client.json.get('user:1', { path: '$' })); // { name: 'Alice', emails: [...], address: {...} }
console.log(await client.json.get('user:1', { path: '$.name' })); // "Alice"
console.log(await client.json.get('user:1', { path: '$.emails[0]' })); // "alice@example.com"
console.log(await client.json.get('user:1', { path: '$.address.zip' })); // "10001"

await client.quit();

@jeffxtang we need to specify the method, PEFT is broader methodology

Lets remove the one without numbers.

are we trying to show the perf improvement as we increased the epcohs?

We need to clarify on what "create_sft_dataset.py" does and if this is a preprocessing step on BIRD dataset?

Overall we need to add more context on the methodology, both for non COT and COT approach.

Lets keep one of them, its just the infra for hosting models, we would not want to compare the two, it will not send the right message in this context. We should provide a way for users to hook up different APIs to the harness here.