If you're traveling by train, there are daily trains... but the more pleasant "Dinaburga" express from Rīga now runs only every other day, departing in the late afternoon and reaching Daugavpils in about three hours (Krustpils/Jēkabpils, on the way, is a worthwhile stop). The Kaliningrad train to St. Petersburg is a possibility for reaching Daugavpils from Vilnius. Daugavpils was on the first railway line in Latvia, Warsaw - St. Petersburg, and the line linking it to Rīga was what made Daugavpils grow and kept Rīga from becoming a backwater. There were very nice trains running soon after the restoration of independence, but the current express is in decline. So, unless you smoke (which you can do at the ends of the railway carriages, not in them) -- take the bus (you'll see more, too; the train rarely runs along the Daugava). There are several buses from and to Rīga every day, and most are very comfortable. The fare is about Ls 3 (€ 4,30 or so) each way, and the expresses take about four hours. If you want to reach Daugavpils from Vilnius, it's best to take a bus to Zarasai, a town near the Latvian border, and then catch a Daugavpils bus from there (there are many). Another option is the Kaliningrad train -- an experience in itself -- but you may end up in the restaurant car. To get _to_ Vilnius from here, you either take the Zarasai bus (and Zarasai is a pleasant place in summer), or ... if you're on a shoestring and adventurous, take the night buses that run to a huge market on the outskirts of Vilnius that sells everything from cheap ball point pens to Kalashnikovs (rumored). Also, there are many buses from Daugavpils to other towns in Latgola and to villages in the countryside. Below is a link to train info, with a picture of the express.
Note -- the Dinaburga Express is again running daily.
There is a pretty good train service to Riga which leaves Daugavpils a few times per day. The most convenient service (to catch aircraft) would be the 06:15 train which gets to Riga at about 9.01. When I used it, it was on time and even had a DVD showing - although the train itself can only really be described as basic. The cost was a very reasonable 4 lats (about £4.00 GBP)
The streetcar lines -- trams -- date only to the Stalin era (probably the only good thing to come from that era...). There are three lines: 1, from the train station to "the Chemistry" (a Khruschev -era area based upon a synthetics plant that now belongs to a French company)... this was once "the Daugavpils Desert" -- sand dunes; 2, from "the Chemistry" to the New Vorstadt (which most people still call the Pervomayskaya, the First of May Microregion -- a newer Soviet monstrosity); 3, from the Fortress to Stropi, reputed to be the sunniest place in Latvia (Latvia not being very sunny), a down-at-heels resort area on a lake. Trams cost Ls 0,20 (€ 0,28). Your interest will probably only be in tram #3, then, to get to the Fortress (and the cemeteries... don't miss those!).
Taxis are plentiful and cheap, though most drivers can't even speak Latvian, much less English -- only Russian. No place in the city should cost you more than Ls 2... but they tend not to use the meter. I've never had a problem with any driver trying to rip me off, and have never heard of such an instance.
But most places you will want to see in the city are within walking distance.