Python Debugger Update

Here I give an update about the progress of the time traveling debugger for Python.

Transcript

The following transcript was automatically generated by an algorithm.

00:00 : not necessarily easy but that is
00:02 : possible
00:03 : is possible yeah
00:06 : okay um that's very cool
00:11 : thanks uh yeah i might i might put this
00:14 : code
00:14 : out there somewhere in case people want
00:17 : to see it
00:18 : um i i want to give a
00:21 : quick update on
00:24 : progress of the python the python
00:27 : debugger
00:29 : okay so there has
00:33 : there has been
00:37 : a lot of work into this um
00:41 : so so at the moment maybe give you a
00:44 : quick
00:45 : demo um
00:49 : what is a file that is
00:53 : interesting
00:57 : okay of course there's the fibonacci
01:02 : uh okay you you run it so this again
01:05 : this is a custom
01:06 : i it's a modified version of python uh
01:10 : i've worked python and i modified the
01:12 : source code
01:14 : to so that when you run a program
01:18 : with this python it spits out a log
01:22 : and at this point i just recently
01:26 : made a change that when when it spits
01:27 : out the log the log will be called
01:29 : the file name dot rewind
01:33 : okay i'm calling this tool rewind
01:36 : or python rewind so it spits out this
01:39 : log and then
01:40 : you can run this recreate that pi
01:43 : script on this log file
01:47 : this pi rewind file
01:50 : and what it does is it generates
01:54 : a sequel light database which is
01:57 : going to be called fip.sqlite
02:01 : and then with this sqlite database this
02:04 : sqlite database the recreate script
02:06 : recreates the
02:07 : previous state of the program
02:11 : and and so the database contains all
02:14 : previous states of the program
02:16 : so we can use our debugger
02:21 : hopefully if it's not broken to uh
02:24 : to step back through the history of this
02:26 : program
02:29 : [Music]
02:31 : i realize the arrow changes direction
02:34 : yeah yeah i i'm still figuring out
02:38 : a um the reason i have this
02:41 : i added this arrow is to show
02:46 : whether you're after the point
02:49 : of calling this function or before the
02:51 : point of calling this function
02:53 : okay um i'm trying to make it more
02:57 : intuitive
02:58 : but i don't think i've totally succeeded
03:02 : but you know it's a work in progress so
03:04 : right now
03:06 : the arrow is pointing up which means we
03:08 : haven't called this fib
03:10 : function yet on this line so you are
03:13 : actually
03:14 : in the line between the last line and
03:17 : this line
03:18 : yeah exactly okay so
03:21 : um yeah so anyway this is the fibonacci
03:25 : program it's the simplest one
03:28 : and it doesn't require any data
03:30 : structures but
03:32 : i've since then i've made it work for
03:36 : lists and dictionaries and sets
03:39 : um other other
03:42 : common data structures for for python
03:46 : uh so
03:51 : let me see maybe try
03:54 : the permutation uh let's try the
03:57 : multiplication persistence one
04:11 : okay there's this program i got from
04:14 : number file the the number file
04:17 : uh youtube channel
04:21 : so if i recreate the past
04:25 : for this one
04:31 : it's actually taking a while and then
04:34 : i'm gonna
04:39 : debug it
04:46 : okay so uh i don't know if i want to
04:50 : spend the time
04:50 : explaining this math problem
04:54 : it's it's okay i think i can so
04:57 : there's a youtube video if you want to
04:59 : watch this one it's a
05:00 : really geeky math thing but you multiply
05:04 : the digits of this number and this
05:07 : number is 2777
05:09 : you multiply all of the digits into the
05:12 : number
05:13 : uh-huh so uh and
05:16 : and if you get to the point where the
05:18 : result is
05:20 : one single digit then you're done but if
05:23 : it's not
05:24 : so first it breaks up into digits so now
05:27 : all the digits are in the
05:28 : list now you see and then it goes
05:31 : through each digit
05:32 : and multiplies it into the result you
05:36 : see
05:39 : now the j is nine now that's one of the
05:43 : last two nines it's
05:44 : multiplying it into this result
05:47 : and that's the result it prints it out
05:50 : and then recursively
05:52 : does this to the new result
05:55 : all right okay this new result it breaks
05:59 : up into digits again
06:00 : and then multiplies them all into this
06:02 : number
06:05 : and then the new result is that and so
06:07 : forth and the
06:08 : the point of all of this is to see
06:11 : what's how many times you can keep it
06:15 : going
06:15 : before there is one single digit
06:20 : so anyway
06:23 : so anyway yeah this is a this is a funny
06:26 : math problem but uh this shows you the
06:29 : debugger
06:31 : works in this case and this case is
06:34 : not super complex uh it uses
06:37 : lists and uh and the debugger is working
06:41 : in this case so
06:42 : um so why it takes a while
06:45 : uh to generate a sql file is it because
06:48 : it has
06:49 : a lot of state uh yeah yeah yeah this
06:52 : is actually really becoming a problem
06:54 : now
06:55 : uh so um
06:59 : last night i i had
07:03 : so as i ran this
07:06 : random recreate script on one of my
07:09 : so so i've been trying various programs
07:12 : that i have written in the past
07:14 : or i could gather from different sources
07:17 : that are actual python programs
07:19 : that people wrote this is one of them
07:22 : this is a ascii drawing program
07:28 : i'll show you quickly
07:30 : [Music]
07:34 : okay so this is an ascii drawing program
07:37 : for
07:37 : for drawing using ascii characters you
07:40 : can
07:41 : you can use the cursor to move around
07:43 : and then you can just type the letter
07:45 : you want oh i love this program
07:49 : so because because my son marty was
07:54 : interested in you know python
07:57 : programming and i i showed him
07:59 : ascii art i made this thing
08:01 : [Music]
08:02 : uh hoping that he would use it but
08:05 : anyway
08:06 : so i made this program i tried to run
08:08 : the debugger
08:09 : on it and the problem is
08:13 : it took first of all there was a whole
08:15 : bunch of
08:16 : various issues various bugs
08:19 : errors that i ran into which
08:22 : some of them i used the test case
08:25 : simplification
08:26 : thing to solve it um okay
08:29 : but then after i got through all of
08:31 : those um
08:33 : it the the recreate script ran really
08:36 : really
08:37 : slowly or just took a really really long
08:40 : time
08:41 : uh and this was last night and and i
08:45 : finally it took so long that i was like
08:47 : okay
08:48 : which which is why i had it um actually
08:54 : i don't want to do it on that because i
08:56 : already created any anyway i ran it on
08:59 : it
08:59 : and i i basically it got so slow that i
09:02 : decided to make it print out the
09:04 : progress
09:04 : and when i printed out the progress i
09:07 : did an estimation to see how long
09:09 : it would finish the whole thing and it
09:11 : would have taken three hours
09:13 : to finish the whole thing i thought yeah
09:17 : i'm about to go to sleep let's keep it
09:19 : running well and see if it will finish
09:21 : while i sleep right okay and then this
09:24 : morning i woke up
09:25 : and then i saw that my
09:28 : my hard drive has ran out of disk space
09:34 : and that was the error that it ended
09:36 : with
09:37 : exactly basically it basically it has
09:40 : generated a
09:42 : 20 gigabyte file
09:46 : uh as as that sequel light database
09:51 : um where is it
09:56 : [Music]
09:59 : this one this is this is a this is a 22
10:02 : gigabyte
10:04 : database that it was generating and it
10:07 : it didn't even finish it was in the
10:10 : middle
10:11 : and it ran out of space in the middle
10:13 : and uh
10:14 : and i was looking at the content of this
10:16 : database and one of the things that
10:19 : obviously jumped out at me is one
10:22 : some of the entries in this object table
10:26 : are extremely large this is a heap
10:29 : this is a representation of a heap
10:33 : of the current state current state of
10:35 : the heap
10:36 : at that point and basically there are so
10:39 : many heap
10:40 : objects in this program that each time
10:44 : any heap object updated i would generate
10:48 : another sort of entry that
10:51 : is like this like with maybe one or two
10:54 : of the things in this map oh because you
10:56 : want to make it immutable
10:57 : yeah yeah yeah so this is an obvious
11:01 : problem and this is so so
11:05 : next step is like uh figure out how to
11:07 : solve this problem
11:09 : yeah i have a question is it possible to
11:13 : um directly insert
11:16 : rows into like a maybe like a database
11:20 : on the internet that has much more room
11:23 : when you go through
11:26 : when you uh inside the thing that
11:29 : inside the python compiler instead of
11:33 : generate
11:33 : a log and then use the log generator sql
11:37 : lite file can we just directly insert
11:40 : stuff because you say you don't go back
11:42 : you don't look
11:43 : back right you don't do like a look
11:45 : ahead or whatever
11:46 : so is it possible to directly just
11:48 : insert while you
11:50 : while the program is running yes
11:53 : it is you're thinking just offload it to
11:56 : the cloud
11:57 : so that yeah exactly it's not a problem
12:01 : yeah exactly well
12:04 : um i mean yeah assuming
12:08 : your computer in the cloud has enough
12:11 : disk space
12:12 : or um well the immediate problem is that
12:16 : it's using a lot of disk space and
12:20 : i think that has to be solved though
12:23 : because
12:23 : it's it's in efficiently using disk
12:27 : space right now
12:28 : right like this is a very very large
12:32 : chunk
12:32 : of string of data and
12:36 : each time any of the heap objects change
12:39 : i'm going to enter another row in the
12:41 : database that's just as long
12:45 : well i guess another potential way will
12:48 : be
12:48 : like using some sort of reference
12:51 : yeah how that will work though it's like
12:54 : you change one string in this long
12:56 : object
12:57 : and then but the rest of it is
12:58 : referencing something else
13:01 : yeah yeah yeah so so this is definitely
13:04 : a
13:05 : naive implementation um
13:09 : what the thing that i was planning on
13:12 : using initially is
13:16 : we can instead of having one
13:19 : gigantic dictionary like this
13:22 : we could separate it into a multi-level
13:26 : map
13:28 : it's like a tree okay we could separate
13:31 : this thing into a tree
13:33 : which is the um there there's been
13:36 : research
13:37 : on how to make multi-level
13:41 : maps based on trees and
13:44 : if it was a tree instead of a map
13:48 : instead of a flat map then what you
13:50 : could do
13:51 : is every time you make a mutation
13:54 : a modification to the tree you keep
13:57 : the notes that don't change
14:00 : and and and then you just you start at
14:04 : the one that you do want to change
14:06 : and then you you basically built
14:10 : a new tree new version of the tree
14:13 : starting at the root and then
14:17 : so the root is going to be new and then
14:20 : but for all the parts that the root has
14:23 : to point to
14:24 : that are don't haven't changed you point
14:26 : to the old ones
14:27 : so it's it's like the if you need to
14:31 : change
14:31 : a node like that's four levels down
14:36 : in within the tree then you have to
14:38 : create a new node
14:40 : for each of its ancestors would have to
14:43 : be new nodes
14:44 : it's like the immutable set right yeah
14:47 : it's exactly
14:48 : it's like the way you do immutable
14:51 : binary trees for example
14:53 : you have to create new nodes for its
14:56 : ancestors but everything else
14:59 : can stay this can point to the previous
15:03 : existing nodes because that those things
15:05 : haven't changed
15:06 : so i could do something like this
15:10 : i'm also considering
15:13 : maybe each of these mappings
15:17 : is just a row
15:20 : in a database
15:23 : and then when you update it maybe
15:25 : there's like a for each
15:27 : heap this is called a heap id
15:30 : but the heap id is pointing to like
15:34 : some object in the heap and then when
15:36 : that object changes
15:37 : then this immutable object id
15:40 : updates right this is a heap id pointing
15:44 : to an immutable object id
15:46 : but you will still have all these ids in
15:50 : this
15:50 : entry that's in
15:53 : here right you don't have the value
15:57 : but you still have the id
16:00 : i don't have oh yeah all of these
16:04 : are only ids the values are elsewhere in
16:08 : this table
16:08 : the the immutable ids are ids back to
16:12 : this table
16:12 : an object inside this table
16:16 : but yeah so i'm gonna think about how to
16:20 : solve this problem um
16:22 : i had a very similar problem when i was
16:24 : working on that tic-tac-toe game that
16:26 : generated a lot of
16:28 : objects
16:31 : but uh yeah i i will
16:35 : i will have fun working on this problem
16:38 : that sounds like a interesting problem
16:40 : yeah
16:41 : it's like it's like algorithm at the end
16:44 : it's like when you wanna
16:45 : um improve the performance and then you
16:48 : just wanna create
16:49 : new algorithm to solve it so that you
16:51 : don't store
16:52 : like repetitive information i guess yeah
16:55 : yeah
16:55 : it is interesting um and you
16:58 : you don't you don't really like i think
17:04 : with performance problems
17:07 : the best approach to
17:10 : solve to uncover performance problem
17:13 : like when you're writing code you're
17:14 : worried about
17:16 : whether this is going to be slow but but
17:19 : when you test it
17:20 : it's usually okay for for the very small
17:23 : data set that you're testing with
17:25 : uh-huh but the way to work on
17:27 : performance is you
17:29 : you just throw some
17:32 : real world data into it to actually get
17:36 : a very very bad scenario and and this is
17:40 : how this
17:40 : has happened uh and it wasn't
17:44 : it wasn't planned is i had no idea that
17:48 : that ascii drawing program would cause
17:50 : this situation
17:51 : but but the reason it caused this
17:53 : situation is not the code
17:55 : in that program itself it's actually
17:58 : because
17:59 : the modules that are loaded
18:02 : by the code like the code uses some
18:05 : modules
18:06 : that cause this and i don't know which
18:09 : modules
18:10 : exactly or maybe it's the combinations
18:12 : of a bunch of the modules
18:14 : they just generate a lot of heap objects
18:17 : in total
18:18 : so by module you just you mean like
18:21 : external
18:22 : packages uh they're they're like
18:25 : they're actually like these modules
18:27 : these modules
18:28 : uh come right out of the box from python
18:31 : python oh
18:32 : okay gotcha but some some of them are
18:36 : directly implemented natively in c
18:39 : but a lot of them are not a lot of them
18:42 : have a lot of python code
18:44 : and my my logger actually logs the
18:47 : execution
18:48 : within those modules as well