Most important is to select the correct firmware. Here smoothieware I guess. Talking with repetier-firmware selected will fail in any case.
Next of course baud rate and port must match. 115200 and 250000 are the most common baud rates. DTR/RTS is normally not relevant for connection, only to reset the board. But is some cases it is relevant. In these try the values HIGH or low in all combinations.