Research First, Paper Trading Second
How to Build an AI Trading Bot (Without Fooling Yourself)
Educational content, not financial advice. Practice with paper trading, not real money.
Searches for "AI trading bot" usually mean one of two very different things: a get-rich-quick script, or a genuine research project that uses Claude or ChatGPT to help write, test, and evaluate a rule-based strategy. This guide is about the second one. It walks through building a research bot step by step: writing one explainable rule, backtesting it with realistic costs, running it on paper for a stretch of time, and reading the resulting scorecard honestly enough to know whether you found an edge or just a lucky stretch of history.
Building an AI trading bot means using Claude or ChatGPT as a coding and research assistant, not as a source of trading signals. The practical sequence is: write down one specific, explainable rule instead of a vague idea; ask the AI to help you backtest that rule with realistic trading fees and slippage included; run the rule forward in a paper trading account for weeks or months before any real money is involved; and read the resulting scorecard, expected value per trade, win rate, maximum drawdown, and out-of-sample performance, skeptically rather than optimistically. Most "bots" that look good never clear this process; the process is the point.
Published
Free lesson · no account neededMeet ClaudeWhy Claude?6 minStart the free lessonKey Takeaways
- An AI trading bot is really a four-step loop: define one rule, backtest it with real costs, paper trade it, then read the scorecard honestly. Skipping any step is how people lose money on a backtest that never worked.
- Claude and ChatGPT are useful for writing the rule in plain language, generating the backtesting code, and explaining the results. Neither one can tell you whether a strategy will keep working, because that depends on data it cannot see yet.
- A backtest without trading fees and slippage is not a backtest of anything you could actually trade. Low-win-rate, high-frequency rules are the most sensitive to this and the easiest to fool yourself with.
- Paper trading exists to test execution and nerve, not just the math. A rule that looked great in a backtest can still fail on paper once real-time data, order timing, and your own impatience get involved.
- The scorecard that matters: expected value per trade, win rate, maximum drawdown, and performance on out-of-sample data the rule was never tuned against. Good results only on the window you built it on is overfitting, not an edge.
- This is education, not investment advice. Nothing here is a recommendation to buy or sell any security, and no result from a backtest or a paper account is a promise about real money.
Copy and Paste
Prompts You Can Use Today
Turn a vague idea into one explainable rule
I have a vague idea for a trading rule: buy when a stock seems oversold and sell when it recovers. Help me turn this into one precise, mechanical rule I could actually code and test, using a named indicator with a specific threshold, a specific holding period, and a specific exit condition. Ask me clarifying questions first if the idea is still too vague to make mechanical, and do not suggest any specific ticker or security to apply it to.
The goal is a rule you could hand to someone else and have them apply it identically, not a feeling.
Write a backtest that includes realistic costs
Write a Python backtesting script for the trading rule I'll describe below, using a well-known backtesting library, and make sure it charges a per-trade commission, models slippage as a percentage of the trade price, and reports results both before and after those costs so I can see how much of the edge they eat. Flag any part of my rule description that is ambiguous enough to introduce lookahead bias, such as using a day's closing price to make a decision earlier that same day. My rule: [describe your rule].
Insist on a before-and-after-costs comparison. A rule that only works before costs is not tradeable.
Design a paper trading log and scorecard
I'm about to paper trade a mechanical rule for several weeks before risking real money. Design a simple spreadsheet log I can fill in by hand for every trade: entry date and price, exit date and price, the rule's signal that triggered it, position size, and the realized profit or loss after estimated costs. Then design a scorecard I can calculate from that log at the end of the period, covering expected value per trade, win rate, maximum drawdown, and how many trades I actually got, so I know if the sample is big enough to mean anything.
Build the log before you start paper trading, not after, so you don't retrofit it to flatter the results.
Red-team your own backtest for overfitting
Here is a trading rule and its backtest results over the period I tested it on: [describe the rule and paste the summary stats]. Play devil's advocate and look specifically for signs of overfitting, lookahead bias, survivorship bias, or a sample size too small to draw a conclusion from. For each concern you raise, tell me exactly what additional test or data split would either confirm or rule it out.
Ask for this after you think the backtest looks good, not before. That's when the temptation to skip skepticism is highest.
Explain in-sample versus out-of-sample testing
Explain the difference between in-sample and out-of-sample testing for a trading rule, as if I'm about to split a dataset into two periods for the first time. Walk through exactly how I should divide the data, what it means if the rule performs well in-sample but poorly out-of-sample, and why tuning the rule's parameters repeatedly on the same out-of-sample period defeats the whole purpose of having one.
This concept is the single biggest gap between a strategy that looks good on a screen and one that might actually hold up.
Set Expectations First
A Research Bot, Not a Money Machine
Most people who search for how to build an AI trading bot are imagining something that doesn't really exist: a script that reads the market and prints profit. What Claude and ChatGPT can actually help you build is narrower and more useful: a disciplined process for turning a trading idea into a rule, testing that rule honestly, and deciding whether it deserves real money. That process is slower and less exciting than the fantasy version, and it is also the only version that has any chance of telling you the truth.
The reason this matters is that markets are adversarial. Thousands of other participants, including firms with far more data and far faster execution than an individual working with a chat assistant, are looking for the same patterns. A rule that an AI can describe in one paragraph is, almost by definition, not some crafty discovery; it's a known pattern that has either already been priced away or that only worked because of a quirk in the specific historical window it was tested on. The honest goal of this kind of project is learning to tell those two cases apart, not finding a shortcut around the work.
None of this means the exercise is worthless. Writing a mechanical rule, backtesting it properly, and paper trading it for real builds a genuinely transferable skill: the discipline of defining exactly what you believe, testing it against disconfirming evidence, and not fooling yourself with a result that looks better than it is. That skill applies far beyond trading. It's also, not coincidentally, the same skill good researchers and engineers rely on every day.
Throughout this guide, Claude or ChatGPT plays a specific, limited role: turning your idea into precise language, writing and explaining code, and stress-testing your own reasoning when you ask it to. It never tells you what to buy, and this guide isn't going to either.
The Core Loop
Four Steps, in This Order
Skipping a step, or doing them out of order, is how a backtest ends up lying to you.
1. Define one explainable rule
Not a feeling, not a vibe, a mechanical rule with specific numbers: what triggers an entry, how big the position is, and what triggers an exit. If you can't write it as a short, unambiguous recipe, it isn't testable yet.
2. Backtest with real costs
Run the rule against historical data with trading fees and slippage included, not excluded. A backtest that ignores costs is measuring a strategy nobody can actually trade.
3. Paper trade it for real
Run the rule forward, in real time, with fake money, for weeks or months, before any real capital touches it. This tests execution and your own discipline, not just the math.
4. Read the scorecard honestly
Expected value per trade, win rate, maximum drawdown, and out-of-sample performance. A good-looking headline number with a bad scorecard underneath is a warning sign, not a green light.
Step One
Define a Rule You Could Hand to a Stranger
The single most common failure point in this whole process happens before any code gets written: the rule is too vague to test. "Buy when the stock looks strong" is not a rule, it's a mood. A testable rule names a specific, named signal, a specific threshold, a specific position size relative to the account, and a specific exit condition, whether that's a price target, a stop loss, a fixed holding period, or some combination.
This is exactly the kind of translation work Claude and ChatGPT are good at: turning a loose idea into a precise, mechanical description, and asking you the clarifying questions you didn't think to ask yourself. Push back when the AI's version is still vague, and keep tightening the language until you could hand the rule to a stranger and expect them to apply it identically to how you would.
Resist the urge to make the rule complicated. A rule with a dozen conditions is harder to test cleanly and far easier to accidentally overfit, since more conditions mean more opportunities to tune the rule until it matches history that has already happened. The simplest version of an idea that's still mechanical and testable is almost always the right starting point.
Write the rule down somewhere before you touch a backtest. If you only keep it in your head, it's very easy to quietly shift the rule's definition later to match whatever result you're hoping to see, which defeats the entire exercise.
Step Two
Backtest With Fees and Slippage, or It Doesn't Count
A backtest tells you how a rule would have performed on historical data. The most common way people fool themselves at this stage is leaving out trading costs: the commission or spread charged on every trade, and slippage, the gap between the price you wanted and the price you actually got, which gets worse the faster you trade or the less liquid the security is.
Ask Claude or ChatGPT to write the backtesting code with both costs modeled explicitly, and ask for results shown before and after costs side by side. For a rule that trades frequently or chases small moves, costs can erase most or all of an apparent edge; seeing the before-and-after comparison is often the fastest way to find out a strategy was never real.
Two other backtest-killers are worth naming specifically. Lookahead bias is using information in the test that wouldn't have been available at the time, such as a day's closing price to trigger a decision earlier that same day. Survivorship bias is testing only against securities that still exist today, which quietly excludes the ones that went to zero or got delisted, flattering the results. Ask the AI to check your rule's logic and your data source for both before you trust a single number that comes out.
Finally, look at how many trades the backtest actually produced. A strategy that only triggered a handful of times over the test period hasn't been tested, it's told you an anecdote. There's no single magic number, but a rule needs enough independent trades for the statistics to mean anything more than chance.
Step Three
Paper Trade Before Anything Else Touches Real Money
Paper trading is a simulated account using live, real-time market data and prices, with no real money at risk. It exists to catch what a backtest can't.
It tests execution, not just math
A backtest assumes you'd have gotten the fill. Paper trading, run in real time, shows whether the signal actually arrives in time to act on, and whether the price you'd get is close to what the backtest assumed.
It tests your own discipline
Watching a rule sit through a losing stretch in real time, with no hindsight, is a different experience than seeing the same losing stretch as a line on a historical chart. Paper trading is where you find out if you'll actually follow the rule.
Give it enough time
A week or two of paper trading tells you almost nothing. Run it long enough, through enough different market conditions, to generate a meaningful number of trades, which for most rules means thinking in months, not days.
Log every trade the same way
Use the same log format you'll use to build the final scorecard. Trying to reconstruct results from memory at the end is how paper trading periods quietly get reinterpreted to look better than they were.
Step Four
Reading the Scorecard Without Lying to Yourself
Four numbers do most of the work in deciding whether a rule deserves more attention: expected value per trade, win rate, maximum drawdown, and out-of-sample performance. Each one on its own can mislead; together they're much harder to fool yourself with.
Expected value per trade is the average profit or loss per trade after costs, across every trade in the sample, wins and losses together. A positive expected value is the baseline requirement for a strategy to be worth running at all; a strategy with a negative expected value is, on average, a slow way to lose money regardless of how exciting any single winning trade felt.
Win rate on its own tells you almost nothing without expected value next to it. A strategy can win 80% of the time and still lose money overall if the rare losses are large enough, and a strategy can win 40% of the time and still be profitable if the wins are big relative to the losses. Look at both numbers together, never one without the other.
Maximum drawdown is the largest peak-to-trough decline the strategy experienced during the test period, and it's the number that tells you what living through this strategy would actually feel like. A strategy with a great average return and a brutal maximum drawdown is one most people would abandon partway through the drawdown, in practice, long before the average return had a chance to show up.
Out-of-sample performance is the strategy's result on data it was never tuned against, tested only after the rule's parameters were locked in on a separate, earlier period. This is the single best defense against overfitting. A rule that performs well in-sample and falls apart out-of-sample was tuned to history, not to a real, repeatable pattern, and the honest conclusion is to treat it as not working rather than to keep adjusting it until the out-of-sample period looks better too.
Where AI Helps and Where It Doesn't
What Claude and ChatGPT Are Actually Good For Here
Both tools are genuinely useful for this project in specific, bounded ways: turning a vague idea into a precise rule, writing and explaining backtesting code, generating paper trading logs and scorecards, and playing a skeptical reviewer of your own reasoning when you explicitly ask for that. Used this way, they can meaningfully shorten the distance between an idea and a properly tested version of it.
What neither tool can do is tell you whether a strategy will keep working. That depends on future market behavior, which is not in anyone's training data, and on competitive dynamics (how many other people are running something similar) that no chatbot has visibility into. Treat any confident-sounding claim from an AI about a strategy's future prospects as something to verify, not to trust, in exactly the same way you'd treat a confident-sounding claim from a stranger on the internet.
A related risk is a model drafting plausible-looking backtesting code that has a subtle bug, such as an off-by-one error that introduces lookahead bias, or a cost model that quietly understates slippage for a less liquid security. Read the code it writes rather than running it blind, and when the result looks unusually good, treat that as a reason to check the code more carefully, not less.
The Skill Here Is Prompting for Rigor, Not Trading
Everything in this guide, turning a vague idea into a precise question, demanding evidence before trusting an answer, and stress-testing your own conclusions, is a prompting skill you can practice on any subject. Nightschool AI's free lessons on research and critical thinking with Claude or ChatGPT start with exactly that.
Frequently Asked Questions
Can Claude or ChatGPT actually predict the market?
No. Neither tool has information about future prices, and nothing in this guide treats them as if they do. Their role here is limited to helping write code, phrase a rule precisely, and critique your own reasoning, never to generate a trading signal to act on.
Is it safe to connect an AI-assisted bot to a live brokerage account?
Treat that as a much later, separate decision from anything in this guide, and only after a long, honest paper trading period and a clear understanding of the real-money risk involved, including the possibility of losing the entire amount risked. This page only covers the research and testing stages, on paper, with no real money.
What's the difference between backtesting and paper trading?
Backtesting runs a rule against historical data to see how it would have performed in the past. Paper trading runs the rule forward in real time, on a simulated account with live prices but no real money, to test execution and discipline that a backtest can't capture.
How long should I paper trade before considering real money?
Long enough to accumulate a meaningful number of independent trades across more than one kind of market condition, which for most rules means months rather than days or weeks. A short paper trading period that happens to coincide with favorable conditions tells you very little.
What is slippage, and why does it matter so much?
Slippage is the gap between the price a rule's logic expected and the price actually filled, caused by the time it takes to act on a signal and by the liquidity of the security being traded. It matters because it quietly erodes an apparent edge, sometimes entirely, and backtests that ignore it tend to look better than any strategy anyone could actually trade.
What does out-of-sample actually mean, in plain terms?
It means testing a rule on data that was set aside and never used while the rule's parameters were being chosen or tuned. If a rule performs well on the data it was built from but poorly on data it has never seen, that is a strong sign the rule was fit to noise in the original dataset rather than to a real, repeatable pattern.
Can an AI write the actual trading bot code for me?
It can write backtesting scripts, data-handling code, and paper trading logs competently when you give it a precise, well-specified rule. Read the code rather than running it unchecked, since a subtle bug, like a lookahead error or an understated cost model, can make a bad rule look good.
Why does expected value matter more than a win rate?
Win rate alone doesn't account for the size of wins versus losses. A high win rate with a few catastrophic losses can still be a net loser, and a lower win rate with well-controlled losses and larger wins can still be profitable. Expected value per trade accounts for both, which is why it matters more on its own.
Is a good backtest proof that a strategy works?
No. A good backtest, by itself, is only evidence that the rule matched the specific historical data it was tested against, which is exactly what overfitting looks like from the outside. Out-of-sample testing and a real paper trading period are what separate a plausible result from a strategy actually worth trusting further.
Build the Habit of Checking Your Own Work
The process matters more than any single backtest result.