Palestra - PyData - Future proof your AI product

Fonte: YouTube

1. 00:06 SPEAKER_01 Hello, everyone, and welcome.
2. 00:07 SPEAKER_01 We're so excited to have you here for another great session.
3. 00:11 SPEAKER_01 Um, we've hope you've had a really great time seeing lots of different sessions.
4. 00:16 SPEAKER_01 I can hope you caught the last keynote.
5. 00:19 SPEAKER_01 Got a lot of great research information from that one.
6. 00:22 SPEAKER_01 And we're in for a really great session again.
7. 00:25 SPEAKER_01 Um, this is Future Proof Your Agents, and Breno Brito uh will give us uh all the great information.
8. 00:33 SPEAKER_01 But before we get started and I hand it over, I just want to remind everyone that please feel free to post your questions or comments in the chat.
9. 00:41 SPEAKER_01 And then there's also a QA uh... uh area as well, if you want to add questions there.
10. 00:47 SPEAKER_01 So we welcome your interaction uh during the presentation, and I'm gonna turn it over to Breno now, and thank you so much for presenting today.
11. 00:56 SPEAKER_02 Hello, everyone.
12. 00:58 SPEAKER_02 Thank you so much for um for being here.
13. 01:01 SPEAKER_02 I'm really appreciate the opportunity.
14. 01:04 SPEAKER_02 And uh yeah, the my presentation is future proof your agents program front with DSPI.
15. 01:13 SPEAKER_02 The idea here is a little bit trying to explain how how why DSPI and uh kind of making a speed run.
16. 01:24 SPEAKER_02 This is a very lots of slides, so I hope I can get you the end so we can answer some questions.
17. 01:31 SPEAKER_02 But before that, my name is Bruno Brito.
18. 01:33 SPEAKER_02 I'm from Brazil Brazil.
19. 01:35 SPEAKER_02 I'm a machine learning engineer, data science scientist.
20. 01:39 SPEAKER_02 Um I'm also uh technical meetup host.
21. 01:43 SPEAKER_02 Also uh I I have a book on Bitcoin because I also work in lots of Bitcoin companies.
22. 01:52 SPEAKER_02 This is also a main point of interest.
23. 01:54 SPEAKER_02 Um, this is my information is always Brandon RB. in X in LinkedIn Gmail, so you can reach me out later.
24. 02:04 SPEAKER_02 Um, so let's get into it.
25. 02:08 SPEAKER_02 Um just table contents.
26. 02:10 SPEAKER_02 We are starting with uh some context, and then we're going to dive deep inside yes by and explain um basically everything.
27. 02:20 SPEAKER_02 So let's get into it.
28. 02:22 SPEAKER_02 So context.
29. 02:23 SPEAKER_02 Um several projects that I've been doing since2023.
30. 02:29 SPEAKER_02 Um they usually lots of people advise not to use any framework, however, I kind of disagree a little.
31. 02:38 SPEAKER_02 There's lots of things that I don't think it's good that we have to re-implement.
32. 02:45 SPEAKER_02 However, every framework that I was using, um, they were opaque.
33. 02:51 SPEAKER_02 Um, and LLM is just strings in, strings out, like most of them.
34. 02:56 SPEAKER_02 There's of course multimodal, but let's focus right now just the the just text ones.
35. 03:04 SPEAKER_02 So if I don't know what's got in what's getting me, it's difficult to steer the model to make it um it turns out the the best things possible.
36. 03:18 SPEAKER_02 So the the most of the frameworks that I tried were very opaque, it's hard to version as well.
37. 03:25 SPEAKER_02 There's um you're building uh writing prompts and testing prompts.
38. 03:30 SPEAKER_02 Sometimes you're testing so so much that you get the worst prompt that you started, but you didn't save because it's kind of it's not good to be uh commiting everything in in Git.
39. 03:47 SPEAKER_02 It's kind of uh hard to do it.
40. 03:50 SPEAKER_02 So it's a it's a not so good thing.
41. 03:53 SPEAKER_02 Also, almost all frameworks they have hard coded prompts.
42. 03:58 SPEAKER_02 So all this um framework set you just like write a line and the the it's ready to use agents, they have lots of right hard coded prompts, and this is this can work best for one model than to another.
43. 04:17 SPEAKER_02 Uh you know, some models work better with if the prompt in the library updates when you update the library, this could be bad for your system.
44. 04:33 SPEAKER_02 It can maybe not work so well.
45. 04:35 SPEAKER_02 Um and also it's hard to inspect evaluates debug uh intermediate steps.
46. 04:41 SPEAKER_02 Sometimes they are doing lots of things.
47. 04:43 SPEAKER_02 I thought so with some companies they are doing this, they the the models are doing so much that they cannot even understand what they're doing, so it's really hard to scale the system.
48. 04:57 SPEAKER_02 And but also prompting itself is a problem inside a prompt, a prompt is basically a huge wall of text, and we have input structuring, uh an inference strategy, output structuring, uh context definition, demo tasks, and other things, but also every time there's a every week we get a new state of the art model, and uh but the models they they require um um different prompting techniques, um most of the models, lot lots of models, they release also a guides to bet... bet better prompt the model.
49. 05:40 SPEAKER_02 So you have to to if you change your your model, you have to like redo all these work on prompting, and also uh they can require new prompting strategies, they can require new structure.
50. 05:54 SPEAKER_02 They can require new structures.
51. 05:56 SPEAKER_02 Even the two definitions can uh we can get new structures, and uh we can we... we need we we we can maybe need to reward and there's always the bitterness and uh applied for prompting that AI may be better pro bet better testor than you so what I the... the main idea here is maybe instead of wasting your system design time rewarding prompts um maybe you don't you shouldn't waste your system design time rewarding prompts you should like work on your system and then you improve your prompts uh you optimized later so like you get a huge prompt how do you even maintain this like this is completely different from every kind of um engineering perspective um so and how this is why I like the SPI so much.
52. 06:58 SPEAKER_02 DSPI stands for declarative self-improving Python.
53. 07:03 SPEAKER_02 I took me years for me to learn, but the documentation is improved since then, and it's way better to understand now.
54. 07:11 SPEAKER_02 And also I'm bringing this digest, and it's hopefully it will be really easy for you as well.
55. 07:20 SPEAKER_02 So basically um the there's three there's four main abstractions.
56. 07:26 SPEAKER_02 We have signatures, we have modules, we have adapters, and we have optimizers.
57. 07:31 SPEAKER_02 So in this picture that I'm showing to you, um we the signature is basically the task or the intent, the input output, and uh we read this task in the inference strategy, which is a module, and the adapter formats the signatures and modules together. to get it into a prompt initial prompt and uh this is basically it and you can use it as it is you can create more modules and stack each other but also there's this optional part where you can optimize your prompts and we'll get there and understand uh how it all work out but it's this is a very um systems engineering uh practice so first of all we get the signature is a task is what you want the model the... the model to do it's the intent that you want it's just inputs and outputs so the the signature is this blue part over here and uh when we put this into uh into a code it's basically this is this small string here that I put tweet in the input and uh happy in the output and I also put uh an optional type that I said that oh the happy is a boolean so that's it I'm going to input the two tweet and I'm going to get happy as output and which is a boolean so it's true or false and uh so that's it you already know how to use DSP thank you for coming to my lecture um I'm kidding there's uh a little bit more but this is the basics that you can already do so many things with it so there's of course a little bit of setup basically one line to choose your model and uh and another line to use this instantiated object so you can re utilise reutilized this uh using lots of different tweets and it gives us a prediction object which is happy equals true so it's uh very simple in three lines you have your product uh your your your your your your your your your your your model working and of course we can get a little bit more complex instead of just like happy boo we can also put confidence float so now we have we input I'm so happy today and we get their our... our response is a prediction object with happy equals true and confidence equals0 point uh0.95
58. 10:29 SPEAKER_02 and uh we can check the types and the types are correct output is boo and the class of uh the output of happy uh the class of happy is bull the and also the type of confidence is float so this oh everything that it's we cannot see on the on this on the this library the spy is everything that you don't need to see it's just making sure that you get uh uh the correct type you get the correct structure you get uh... uh a good structure every time the uh the types that you want every time so that's basically you don't need to think about it so let's get the little bit more complicated a little bit more complex um so let's say um I want to I'm getting an email and I want to get a list of tech uh tasks that email send me so I put uh my input is today which is a string uh email which is a string and then task which is a list of strings and then the deadline which is a string uh nice thing that in this example is that by how you define uh how you name your variables, for example, deadline YY M DG, the model will out to output put this to you.
59. 12:03 SPEAKER_02 Look here that I didn't say anything about uh I I didn't uh made any scaring I could use Pydentic, for example, and say that I want exactly this output, but I didn't.
60. 12:18 SPEAKER_02 But since this is going inside the model, uh the the model, of course, will understand the deadline y mm d is that I want a date, which is the deadline and uh a year, month, and day.
61. 12:36 SPEAKER_02 So that's exactly what I get here.
62. 12:39 SPEAKER_02 Uh already formatted as I wanted.
63. 12:42 SPEAKER_02 Of course, so we can uh make another uh... uh validation stage, but for most cases uh, especially in the beginning, it works wonders, and it's very very good already out of the box.
64. 12:59 SPEAKER_02 And uh okay, but if I have lots of inputs and lots of outputs, uh I would have to have a uh a really big string here, and we are getting into the first problem again, and of course not uh we can use a class, a signature class, and uh we can structure a little bit more and have uh... uh more information in this signature in this task, what you want.
65. 13:29 SPEAKER_02 So basically the doc string is part of the code, so the doc string is an instruction that goes into the into your model, it's important for you to know that there's also input, which is just an input field, it's it's very it's basically the same as uh Pydenic field, and uh so the attachments here we can have a more complex um type.
66. 13:57 SPEAKER_02 So for example, here is a dictionary of strings in mapping into DSP.image,
67. 14:05 SPEAKER_02 so we can input an image here.
68. 14:09 SPEAKER_02 Uh we don't need to make lots of different conversions, so this works also for images and for audio.
69. 14:16 SPEAKER_02 There's types for that, and you can easily work with them without using lots of boilerplates.
70. 14:24 SPEAKER_02 It what I like from DSP is that it abstracts all of your boilerplate boilerplates, and of course the outputs, so we get a list of events, we can define the events in another uh as a class somewhere else and use it, and also the action items with uh with the action, and also the the priority, which I put here as literal.
71. 14:56 SPEAKER_02 So I will also uh I will always get P0 P1 and P2.
72. 15:01 SPEAKER_02 So this is this also steers my model to specific outputs and uh specific types, and I can for example if I'm doing a classification, I can use literal and use the classification here, so I know that it will never invent a new classification for me.
73. 15:20 SPEAKER_02 It will always output this type here that I just got it.
74. 15:25 SPEAKER_02 So uh you could put an inum, for example, and so this is amazing for me, and of course, there's uh this desk, which is description, so we can uh explain a little bit to the model what your what is the input, what you want for the output, uh so you can have a little bit more. specific um details on this.
75. 15:50 SPEAKER_02 Okay, so that was everything on signatures, uh basically almost everything.
76. 15:57 SPEAKER_02 And uh what are modules?
77. 15:59 SPEAKER_02 Module are inference strategies, but also building blocks.
78. 16:03 SPEAKER_02 Um what is an interest struct strategy?
79. 16:07 SPEAKER_02 So we were so the it's the this blue part and this yellow part.
80. 16:13 SPEAKER_02 Uh I just showed you some code that use this DSP.prodict
81. 16:19 SPEAKER_02 and struct events.
82. 16:22 SPEAKER_02 So predict is the most basic simple module that exists, uh, but it's also an infrastructure strategy, which is basically not using any inference strategy, you just like prompt your model, and that's it.
83. 16:38 SPEAKER_02 Uh however, you can use other kinds of infrastructure like chain of thoughts.
84. 16:44 SPEAKER_02 So the model will use the chain of thoughts.
85. 16:48 SPEAKER_02 Uh you can use this batch module that is a little bit more complex, um module that will get these email threads and and um running parallel models of you can you can run as many as you want depending on your computer and what you want, but you can run lots of uh things in parallel.
86. 17:15 SPEAKER_02 So this is tractor is your is your instance, and you can use batch and uh to to use lots of email threads instead of just one at a time.
87. 17:27 SPEAKER_02 Also, we can use react, so you change just one line here, one methods, and now you got an agent.
88. 17:37 SPEAKER_02 So you use here your react, you can give tools, tools are basically Python. functions, you don't need to wrap them in any thing, you don't need to put them inside a uh decorator or in a special function or do anything, just write a Python function and you put here and it's a tool you can use it.
89. 18:01 SPEAKER_02 Best of n it run n times this is three, for example, you run extractor n times, and then you it gets you the the the best one according to this rear reward function that you can just invent here is just another python function, and uh so each and every one of these is a module prediction of thought react best of n so and uh it's very it this is a very simple uh thing, but you can do a little go a little bit more complex, just like we did in signature, we can use a class uh in from from from from from from from from from from from the SPY module, and uh if you're looking at this, if you use PyTorch, it's very similar to what you use before in PyTorch.
90. 18:58 SPEAKER_02 You have uh init class uh uh init methods and uh forward methods.
91. 19:04 SPEAKER_02 So basically you declare what you are going to use in it, and then you use it in forward, and uh so this is the hop methods that use this um thing here, and it gets claim here.
92. 19:21 SPEAKER_02 You can see that in forward it uses claim, so you just use hop claim equals to, and you pass what you want, and uh it will do this logic over here and give you back what you want, and uh this is very cool because when you're we are in the next phase. um in the optimizer section.
93. 19:45 SPEAKER_02 We will see that we don't need to optimize every prompt in every um for every signature.
94. 19:54 SPEAKER_02 We optimized the program.
95. 19:56 SPEAKER_02 So we don't want uh overly optimized task.
96. 20:02 SPEAKER_02 We want uh an optimized system, we want to everything to work really well considering the other parts to get the best response possible.
97. 20:15 SPEAKER_02 So this is what uh so when we optimized we... we can optimize this high order module here that will optimized everything inside for you automatically so you don't need to think about saying this is this is very good.
98. 20:33 SPEAKER_02 Um okay so but I was telling you about like how the other ones are bad they are opaque we cannot see what the prompts are so let's take a look into adapter so we understand what is going on what is the actual prompt so we have signatures that we wrap them in modules and uh we use adapters to format them into prompts.
99. 21:02 SPEAKER_02 Uh so when we don't say any when when you the default adapter is the chat adapter, when you don't define any adapter, this is the one that it's going to use.
100. 21:14 SPEAKER_02 So for example, using this context manager here, we can use that uh old tweet sentiment uh program, really simple one.
101. 21:24 SPEAKER_02 I can uh ask it to generate some some some some some some some some some some some kind of response, and I can use this yespy.spect
102. 21:33 SPEAKER_02 history and see what went inside the the model and what the model um spit it out so this is uh what the model formats it so i just insert tweet goes to happy which is a boo and hope this which is a float and it creates all this system message and all this user message here and we get this response here and that's how it can parse and everything and have it structured for us and uh I'm not going to spend a lot of time here because it's kind of simple and you don't need to actually uh look too much at it because we're going to optimizers and you understand why but there's a JSON adapter so it will try to format things in JSON uh some models work better in JSON some other models work better in XML so you can use it to give you this format here um but also there's this very interesting one with a which is a two step adapter it will format not only the system message but also the user message uh with instructions so it's more um um it it usually adheres to your to your prompt better if you're doing some very complex thing uh sometimes you need to use the two step adapter and it will work um instead of the other ones so this is uh it's a maximum hearing force let's say like this but also you can create your own uh adapters but it's a little bit complicated at the moment but um uh uh usually the the those one work so okay we uh we... we we're now in optimizer phase we are
103. 23:36 SPEAKER_02 So is there a phase we already have a program that it work that works that makes uh implements all all of our uh... uh our logic in our system.
104. 23:47 SPEAKER_02 So everything is working, but we want it to work even better.
105. 23:51 SPEAKER_02 Uh maybe we'll we want to change models and and try a little bit better, uh change strategies from predict to chain of thoughts and other things like this.
106. 24:03 SPEAKER_02 So one way to do it is uh I believe maybe the best way to do it is going to optimizers, and uh first of all, um let's explain just a little bit what we done we need.
107. 24:18 SPEAKER_02 We we we need of course a data set, which is basically an example.
108. 24:23 SPEAKER_02 Uh we it's the spy dot example, and we put the inputs and the outputs uh here, and we define we give to the model what are the actual inputs, and after we have this data set, of course, we need to evaluate.
109. 24:40 SPEAKER_02 We need if we want to optimize, we need to evaluate how it was and how it is now after optimizing.
110. 24:48 SPEAKER_02 So there's this evaluation um part of the library where you can put the uh... uh your set.
111. 24:56 SPEAKER_02 Uh you can do it in parallel, and uh you give a metric.
112. 25:01 SPEAKER_02 Again, the metric is just a Python function, it's very very simple.
113. 25:07 SPEAKER_02 Um so this is a simple, very simple LLM as a judge.
114. 25:13 SPEAKER_02 I put the judge here, and I I just use it to output this uh uh if it's similar and how confidence it is.
115. 25:23 SPEAKER_02 So this is just uh a uh simple example, but now let's go to the actual uh to the to the actual optimization strategies.
116. 25:33 SPEAKER_02 So the simplest one. it's bootstrap few shot.
117. 25:40 SPEAKER_02 Uh everyone that knows prompt uh engineering knows that the simplest way to improve your prompt is give some is to give some demo some examples, which is known as few shot prompting.
118. 25:56 SPEAKER_02 So this is an automatic way to do a few shot prompting.
119. 26:00 SPEAKER_02 It selects the best.
120. 26:05 SPEAKER_02 And it selects the best examples to use in your in your prompts, and uh it uses it compiles and adds as a demo inside your program.
121. 26:18 SPEAKER_02 Um the second one is a micro is a little bit more complex.
122. 26:22 SPEAKER_02 I won't get into the detail, I just want you to know that it exists and you can then study it later, but basically it's a multi-prompt instructional proposal optimizer, so it gives a lot of lots of different uh prompts and instructions, and it proposed lots of prompts and and it tries it out, and there you can give a teacher then will help to steer um the the how it will improve your prompts.
123. 26:52 SPEAKER_02 Also there's FIMA, which is fantastic introspective mini bash sense, which is something like the the other one.
124. 27:03 SPEAKER_02 Uh uses some uh a little bit of base, but also you give some you give your program, you have the student, you can have a teacher, and then uh some metric, and it will generate lots of um instructions, lots of prompts, and uh try the best ones according to your metric.
125. 27:25 SPEAKER_02 And gip uh lots of people are talking about gip uh um it's also the same thing, but it's uses a genetic pareto.
126. 27:32 SPEAKER_02 The genetic Pareto.
127. 27:34 SPEAKER_02 But also you should read about it in the documentation in the website later.
128. 27:40 SPEAKER_02 But what I think it's very very interesting, not saying that this is the best thing, but it's if we want to optimize.
129. 27:53 SPEAKER_02 We can uh make a fine tuning because fine tuning is the same.
130. 27:58 SPEAKER_02 We give examples, uh inputs and outputs, and we run some some some some some some some some some some some codes and optimize our system.
131. 28:06 SPEAKER_02 So we can do this uh for uh this this we can use this boost bootstrap fine tune and actually have a fine tuning um update your weights in of your system not only the your prompt so you... you can use both for example you can update your the the weights and also the prompts but also you can use for example GRPO which is which was made very famous by DeepC uh it's it seems very big, but actually it's just you define your variables, you instantiate the class, and then you run the optimizer.
132. 28:49 SPEAKER_02 It's the same, it's it's simple.
133. 28:52 SPEAKER_02 The the complex part is just the variables that you need to to work it out, but it's it's it's inside the library, you can just run it, and uh so it your you have inputs and outputs, you can you can do whatever you want it, you're not restricting just to prompting.
134. 29:12 SPEAKER_02 So this is something that I find very interesting from JSPI.
135. 29:16 SPEAKER_02 And of course, don't forget to save your program, you can use save and load.
136. 29:21 SPEAKER_02 Um so with JSPA, you can easily keep up with the latest stack.
137. 29:27 SPEAKER_02 You just change a line, you change the strategy, you change the test, you add a little bit more, and you can version it, you can uh keep it future proof, just like you maintain software, uh, you can maintain your AI software without needing to uh have all the wall of text that is so complicated to maintain.
138. 29:51 SPEAKER_02 So that's it.
139. 29:52 SPEAKER_02 Thank you so much.
140. 29:53 SPEAKER_02 If you want to try uh this this code, uh I have it here in this link that you can follow.
141. 30:01 SPEAKER_02 You can follow me on on on Twitter, send me an email, and I will try to answer if you do it.
142. 30:09 SPEAKER_02 Thank you so much.
143. 30:11 SPEAKER_01 Thank you, Breno, for this great presentation.
144. 30:14 SPEAKER_01 We really appreciate it, and I hope everyone has a wonderful day, and please continue to join us for sessions.